More splunk fun...

I've started setting up summary indexes. I take results and put them in a second index for reporting. First you have to create the new index, mine's called "dashboard_summarize". It will require a restart of splunk, just so you know. Next up, the ugly query:

search = host="srchqenmana*" (source="/usr/local/tvs/apache-tomcat/logs/qlogger/*" NOT source="*.gz") "<A9_Request" AND NOT ("FFFFFFFFFFFF" OR "000013ED3AEB" OR "Agent.007") | lookup Market_by_Controller_ID Controller_ID as Controller_ID OUTPUT Market as Market | eval QueryFirstTwo=substr(TextQuery,1,2) | transaction MAC, QueryFirstTwo maxspan=5m maxpause=1m delim="," mvlist=TextQuery | eval LastQuery=mvindex(TextQuery, -1) | fillnull value=0 forward | eval MAC="costtimequalityscope".MAC | eval MAC=md5(MAC)|stats count(LastQuery) as QueryCount by LastQuery, Market, Controller_ID, StreamingServerID, forward | fields QueryCount LastQuery Controller_ID StreamingServerID Market forward |collect addtime=true index=dashboard_summarize

Yikes! Lets break that down a bit. First up we have the sifting portion of the query. Basically search terms that rule data pieces in our out:

host="srchqenmana*" (source="/usr/local/tvs/apache-tomcat/logs/qlogger/*" NOT source="*.gz") "<A9_Request" AND NOT ("FFFFFFFFFFFF" OR "000013ED3AEB" OR "Agent.007")

Next up we have some data lookups. We take the numerical ControllerID and map that to a human readable market name like 'Salt Lake' or 'Bucks County'.

lookup Market_by_Controller_ID Controller_ID as Controller_ID OUTPUT Market as Market

Next we start doing calculations, conversions and transformations of the data. We'll stanza by stanza this part:

eval QueryFirstTwo=substr(TextQuery,1,2)

Eval a field called 'QueryFirstTwo' to the first two letters of the string TextQuery using the substr function

transaction MAC, QueryFirstTwo maxspan=5m maxpause=1m delim="," mvlist=TextQuery

This little gem is a beauty. I wish I could take credit for what the Splunk consultant did there. Basically we define what a single user search is here by defining what a transaction is.. We do not count just the simple submission of a request, because we do live updating of search results after two letters. So if you were searching for the show HOUSE, with live updating you would make a request for HO, HOU, HOUS, HOUSE at every key press. That's great if your just measuring raw throughput, not not a valuable business data point. If everyone is searching for a really long search terms like SUPERNATURAL your usage stats would be skewed. So we roll those up into a single transaction by setting some parameters. First, we time box it at 5 minutes. Second we only allow for a 1 minute pause. Sure there are edge cases where you may exceed either of these time boundaries but it should be a wash over all. Further the MAC address and the first two letters of the search must also be the same. This lets us have typos later on. So if you did HOUU the HOUS because HO would match, it's still in the same transaction. And the last little bit, mvlist=TextQuery says to make a multi-value (or array) of TextQuery values used in this transaction. In my example the list would have ("HO", "HOU", "HOUS","HOUSE"). This comes up in our next stanza.

eval LastQuery=mvindex(TextQuery, -1)

If you look up mvindex and it's syntax, you see that we're setting the field LastQuery to the last entry in the list. In my example, LastQuery=HOUSE. Side note: the page linked for mvindex is titled 'Common Eval Functions' according to the URL. I'd hate to see the uncommon ones.

fillnull value=0 forward | eval MAC="salted".MAC | eval MAC=md5(MAC)

I'm grouping the next three stanza's together because they're doing similar things. If the field named "forward" is null, set it to zero. Next we add a salt to the MAC address to obscure/anonymize it. The MAC (much like an IP address), while not directly identifying an individual is sensitive just the same, and needs to be hidden, so first we add the string salted to the current value of MAC. Think of this like a password or key. Next we convert the string+MAC value to the MD5 HASH of that string. So 000013ED3AEB becomes salted000013ED3AEB which becomes ce431f1c1a634337ca1cdcde78a1d15f. Now if someone knows someone's MAC address and does echo -n "000013ED3AEB" | md5sum to try and figure out their new obscured value, they can't because they don't know the SALT. And because the salt can be of arbitrary length, brute force isn't effective. So it's reasonably protected if for some reason the data needs to be shared with non-trusted parties.

stats count(LastQuery) as QueryCount by LastQuery, Market, Controller_ID, StreamingServerID, forward, MAC

This one is fairly straight forward. Get the number of times the search term was searched, organized by Market (which we looked up in a table before) and Controller_ID, StreamingServerID, and the value of forward (which are app specific fields that only has meaning to us) The why of this is coming up.

fields QueryCount LastQuery Controller_ID StreamingServerID Market forward, MAC

Next we want to take the fields listed above and output them in the search results (why is next).

collect addtime=true index=dashboard_summary

Lastly, we collect this data and store it into an index called 'dashboard_summary'. What we're doing is making a roll-up of searches and weeding out all the cruft that isn't needed to make the reports or dashboards. Further because we've scrubbed sensitive data, we now can let a larger audience view the data by giving them only permissions to this new index. Because the index is lean and mean, dashboards and reports are several orders of magnitude faster than going against the raw data. Further we've pre-paid a lot of calculation expense with the eval's and transaction logic.  
Now I have an index to do my reporting out of that's much faster than the raw queries against all the data.

Working with Splunk

I've been doing a lot of work with Splunk lately. Splunk is a powerful and flexible indexing tool. It slurps up log files and data and makes them searchable. I think the real power of Splunk over a lot of other log management and searching tools is it's ability to search across multiple servers for the same time period. Another powerful feature is it's ability to do field extraction. So when a log file says "IP_Address=10.11.12.13" you can do field related searches like "AND IP_Address=10.11.12.13" or more powerfully "NOT IP_Address=10.11.*"

Fields are where I'm spending a lot of my time lately. In our current search and discovery platform we have lots of fields with interesting values from people making search requests. We have values such as channelmap, controllerID, MAC, TextQuery and a few other interesting values. Because we have these interesting field values and Splunk extracts them for us, we can generate very interesting usage reports. Such as number of unique users, users per market, etc. And because we have a relatively closed set of users, we can produce interesting numbers like the percentage of users our platform. Powerful stuff.

Anyway, I hope to write up some of my more interesting uses of splunk in the future.

Ping gets more useful

I didn't get Ping from Apple.  What good was it?  I mean even if you had all the artists in the iTunes 'verse online and updating, so what?  Well with the recent update to iTunes, it's starting to become more of what I thought it should be.   They finally added some features that iLike.com and last.fm had all along.

iTunes 10.0.1 makes it easier to share your favorite music with your friends on Ping. You can now Like or Post about music right from your iTunes library. You can also easily see the recent activity of a selected artist in your library, or of all artists and friends you follow on Ping using the new Ping Sidebar.

So now it gets more integrated into the core App, you don't have to visit the iTunes store to view Ping info.  It has a nice side bar like ilike.com (a service I've stopped using some time ago).  You get a facebook/twitter style timeline of what others bought, followed, commented, etc.  All of this is nothing new in the 'social media' world.  Nor is it done innovatively or particularly well.  I think it's too much of a 'me too' move by Apple.  Simply putting Apples brand and market presence behind it isn't enough.  Google Wave?  Microsoft Zune or Bing? The product still has to be good and useful.  So far Ping seems to be neither to me.  It also seems to be solving a problem nobody has.  Facebook and Twitter tell me all about what my friends are up to.  Do I want to go yet another place to see what they're listening to?

Hopefully Apple will roll out a regular string of improvements to the service.  In typical Apple fashion they're not rushing into this.  They released the first, fairly crippled version a few weeks ago.  They've already released the first update.  With any luck they'll release another before the years end.  I'd like to see it incorporate the Genius suggestions some way.  It'd also be interesting to give out some kind of badge or award for listening and rating.  They also need to improve the way you find people to follow and they suggestions they generate. Some other interesting features would be to suggest a 'mood' a person is in based on the music they listen to.

Now on to the wild speculation based nothing except my wishful thinking.  Ok, suppose they get cool new features into Ping.  So what?  Is it enough to reach the tipping point?  I doubt it.  But what if it's part of a bigger plan?  What if Ping goes beyond iTunes and takes the next logical step and gets integrated into iPods, iPads and iPhones?  Now it's more than music.  But that's not enough. I can already twitter and facebook on those devices. What if it extends further to the AppleTV? Now it's about what I watched in addition to what I listened to.  Now it's getting interesting.  That's one niche that hasn't been filled by cable or FiOS.  Tivo, Roku and Boxee are headed there, but they're one dimensional.   Watch a great TV show, then comment about it to all your friends.  Even better if it could be done while watching.  Ping your buddy while watching; 'Hey, I know you'd do exactly what Wolowitz did with the robot arm!'  Now Ping becomes something more than a copycat app.

Anyway you slice it, Apple has a lot of work ahead of them if they hope to turn Ping into another reason to use iTunes and the Apple eco-system.

Need some new fitness gadgets...

So I have a GPS on my bike, an older 12-channel eTrex. It has it's problems. It looses signal too often in the city so my stats are off a bit (I've was at an elevation of -5 feet) for a mile or so today. So I'm looking for new gadgets to use for this.

So far I've come across the ANT+ system by Digifit and I think it does exactly what I want. Since I use my iPod when I ride anyway, it's one less gadget to carry around. Amazon sells it for ~$80. Since ANT+ is a relatively open system, there's multiple vendors making gear for it:

• Adidas sensors and devices (ANT+)



• CycleOps sensors and devices (ANT+)



• Garmin sensors and devices (ANT+)



• Quarq sensors and devices (ANT+)



• Spinning® / StarTrac



• Tanita weight scales (ANT+)



• Timex sensors and devices (ANT+)



• Wahoo sensors and devices (ANT+)

Plus a host of others. So I can add a Garmin speed sensor and a heart rate monitor. If I go to the gym I can get the info from the StarTrac treadmills. Not sure I'll go as far as the Tanita scales though.

Boxee pre-order available (why would you?)

So Boxee is now availble for pre-order, according to the press-release I was emailed:

D-Link has signed up Amazon http://amzn.to/theboxeeboxbydlink (in the US) and Best Buy http://www.bestbuy.ca/boxee / Future Shop http://www.futureshop.ca/boxee (in Canada) as exclusive pre-order partners for the Boxee Box.

 

The highlights they point out:

• it will have access to more HD content than its PC cousin



• no need for keyboard/mouse in the living room or running a 10ft cable to connect your laptop



• it’s beautiful, though a bit pointy in parts : )

 

Beauty is in the eye of the beholder I guess, I don't care for it myself:

It's OK I guess, but that needs to sit next to your TV. It's only 4.5" x 4.5" x 4.6" so it's not too big, you could probably squeeze it on top of your DVR or cable box, but the area around it becomes unusable space for me. The one thing I loved: The remote. It has a basic 4-axis control pad with a 'select' button, a play/pause and what looks to be a power button. That's all run-of-the-mill stuff. The cool stuff is on the flip-side. Flip the remote over and it has a full keyboard. If you've ever tried to search on an AppleTV, Tivo or Cable remote you know how huge this is. No word so far on if it supports Hulu. The big thing for me is the price. $199 per unit. It's $100 more than the new Apple TV and $140 more than a Roku. Unless they have Hulu, I think the Boxee box is DOA....

 

My kids will never have a 'must see TV' night

There was a time when Thursday nights were 'must see TV' nights. Friends, Seinfeld and ER made a pretty compelling night of 'must see TV' as the slogan went. A significant portion of the nation would be sharing the same experience Thursday nights. I remember getting up early on a Saturday, even though it wasn't a school day because that's when the good cartoons where on. I doubt my kids will ever have that notion or experience. Between DVR/TiVo, On Demand broadcasts, Web delivery and AppleTV there's not as big a driver to sit down at a scheduled day and time and watch.

For the past few years we only had terrestrial broadcast TV and an AppleTV in the house. One time, while watching Arthur on PBS, Renee had to go to the bathroom and was jumping up and down demanding we 'pause it!', not understanding that not all TV shows were like the AppleTV version. A few months back we added Cable to the house and a DVR unit, so now we can pause live TV, further blurring the distinction between the on demand vs on schedule showings. She doesn't understand why she can't watch Zula Patrol any time she wants and has to wait until 7:30 to see it. She's convinced its something that I'm not doing for her, and not a case of it not being available On Demand or via AppleTV. Further there's only one episode to watch and when it's done it's done. With all her other shows there's always another episode, so she asks to watch another Zula Patrol, and i have to tell her no. From her perspective it's no different then saying she can only watch one Super Why. We have more, I'm just not allowing her to watch them. Except in this case, we legitimately don't have more to watch. Kind of works in my favor I guess, there's no chance I'll cave in and let her watch a 2nd one.

As more and more entertainment options become less tied to the providers schedule and less tied to the TV as the only way to watch it, the notion of a good night for TV will wither away. There will still be some notion of scheduling, but it'll be the date and time it's put on the distribution network. It probably won't be the same though. Even for me it's not quite the same. I love CBS's monday line-up, but I don't think I've watched any of them at there actual broadcast time in 2 or 3 years. I don't think TV's dead or going away. As Randall Hounsell put it, TV is still "a lean back experience." People will still want to get someplace comfy and be immersed in a world that isn't there own.

Back when CD's were the norm, but vinyl records were still around, my over-used joke used to be that my kids were going to be asking me "Dad, how do we get this big black disc into the CD player?" Now I'm not so sure they'll even remember what a CD is. Never thought the same would happen for TV. The Qwest commercial from the late-90's is finally coming true.

A tired man goes into a cheap motel in the middle of nowhere and asks about amenities. When he asks about entertainment, the girl responds "all rooms have every movie ever made in any language anytime day or night." It'll probably be 20 years after the fact, but it's coming.

Getting there from here

I've started this rambling post four or five times. Deleted it every time. So I've decided to take a different tact. Rather than writing a long post that says everything, I'm going to write pieces of it. This way I'll actually get something written. As the Japanese proverb goes "Vision without action is a daydream. Action without vision is a nightmare.” All too often I've run into folks who are long on vision but short on action. A few times I've been in organizations that are restructuring to better cope with the current environment. The ever present 're-org'. One of those was the transformation into a service delivery organization. Which was a good idea and there was good vision behind it. The action is where the idea died and cost people their jobs.

I've had this picture hanging around for a few years. I stole it from an issue of eWeek. It was from one of those articles that is really nothing more than a advertisement in essay form.

This picture does a pretty good job at explaining what I mean. Although it's not the 'buzz word' it used to be, being a 'service based organization' was the goal of a lot of IT organizations. On paper it looks great. It can be an effective way to run an organization. Unfortunately the trick is getting from where you are to where you want to be. It's been all to common for organizations to 'green field' the new way of doing things and make a sweeping change to transform into the desired structure in the shortest possible time frame. And it's usually a disaster for the first two quarters. There's a lot of uncertainty on how things get done or who does them. Process bottlenecks creep up everywhere. There's inconsistency in implementation between teams. And while all of this is going on, real work needs to get done to keep the business going. After a while people start to revert to the old way of doing things or a hodge-podge in between the old and new.

 

The step that gets missed is the transition and how much transition can be achieved in one fell swoop. If you're currently a 'turmoil' or 'reactive' organization and you want to be a service-based organization, it's unrealistic to jump right to the end state. With out learning the lessons that come with being reactive, it's difficult to be proactive. If an organization doesn't have a solid proactive foundation, it can never be service based. Worse yet, there are budgetary considerations that go along with crossing over from one level to another. Software and hardware tools are often needed to achieve the desired state. Although often over looked in the planning stages, it's possible to make up that budgetary gap. Another gap that's overlooked is the people side of things. I have never seen an organization budget staff time and overhead to these types of changes. It's always expected to be done in the margins after a one or two hour 'training course' that typically just reads the new process aloud to everyone in attendance. No attention is paid to how to get the staff to the end goal. No real-world examples provided for how things should work. No governing authority to turn to for guidance. No one to find parts of the organization that our floundering in the new process/structure and pitch in and help them through it. Proud in their new organizational structure and plan, leadership pass it down the chain, with implementation left as an exercise to the reader.

 

So as I try to manage my team, I've tried to utilize some of the failure lessons I've learned. I don't make broad sweeping changes if avoidable. There always needs to be a balance of course, you don't want to make hundreds of small course corrections when a few larger ones will be as effective, but I lean towards the smaller changes. I plan in the overhead. If I'm going to add new processes or procedures to my staff's duties, I adjust time expectations accordingly. An example would be our post-mortems on outages. I wanted to change how that was done. It should be a 30 minute meeting, but because people were new to it, the first few where schedule for 60 to 90 minutes and we brought in lunch. Walk everyone through the new process a few times. Going back to the post-mortem example, we talked openly about the process and the actual problem in the same context. Giving people a new process with out a concrete example to work with leaves things to interpretation, and you'll get as many interpretations as you have staff members. By walking through it a few times with everyone they all here the same questions and my answers to those questions. It's not perfect or without flaws, but it seems to be working.