Brocade oddness today...

Came across some unusual brocade errors today. And since google turned up nothing, I suspect it's pretty rare.

When I telnet to the box, I get the following:

Connected to 10.20.30.40.



Escape character is '^]'.



libipadm error: client connect failed /tmp/ipadm-g-login-23293-d6ba659370229

ipadm do_backtrace connect failed pid 23293, name login

/fabos/lib/libipadm.so.1.0[0xf6720c0]

/fabos/lib/libipadm.so.1.0[0xf6723a8]

/fabos/lib/libipadm.so.1.0(ipAdmLocalAddrIdGet+0x4c)[0xf6726c0]

/lib/security/pam_fabos.so[0xfd889e4]

/lib/security/pam_fabos.so(pam_sm_authenticate+0x25c)[0xfd891c8]

/lib/libpam.so[0xff8c7d0]

/lib/libpam.so(_pam_dispatch+0x2b0)[0xff8cd30]

/lib/libpam.so(pam_authenticate+0x90)[0xff8f17c]

/bin/login[0x10003254]

/lib/libc.so.6[0xfdd8930]

/lib/libc.so.6[0xfdd8a34]



Fabos Version 5.3.1



Password:



BottomSwitch:admin> ipaddrshow

libipadm error: client connect failed /tmp/ipadm-g-ipaddrshow-23398-d6bb020d2dce6

ipadm do_backtrace connect failed pid 23398, name ipaddrshow

/fabos/lib/libipadm.so.1.0[0xf8ca0c0]

/fabos/lib/libipadm.so.1.0[0xf8ca3a8]

/fabos/lib/libipadm.so.1.0(ipAdmLocalAddrIdGet+0x4c)[0xf8ca6c0]

ipaddrshow(main+0x44)[0x10006018]

/lib/libc.so.6[0xf75e930]

/lib/libc.so.6[0xf75ea34]

ipAdmLocalAddrIdGet() returned 21

BottomSwitch:admin>

The errshow gives me the following from weeks ago:



2008/05/15-02:40:35, [RAS-1001], 137,, INFO, SilkWorm48000, First failure data capture (FFDC) event occurred.

2008/05/15-02:40:41, [RAS-1001], 138,, INFO, SilkWorm48000, First failure data capture (FFDC) event occurred.

2008/05/15-02:41:03, [TRCE-1001], 139,, WARNING, SilkWorm48000, Trace dump available (Slot 6)! (reason: FFDC)

2008/05/15-02:41:03, [TRCE-1004], 140,, WARNING, SilkWorm48000, Trace dump (Slot 6) was not transferred because trace auto-FTP disabled.

2008/05/15-02:41:05, [TRCE-1001], 141,, WARNING, SilkWorm48000, Trace dump available (Slot 6)! (reason: FFDC)

2008/05/15-02:41:05, [TRCE-1004], 142,, WARNING, SilkWorm48000, Trace dump (Slot 6) was not transferred because trace auto-FTP disabled.

2008/05/15-02:42:08, [RAS-1001], 143,, INFO, SilkWorm48000, First failure data capture (FFDC) event occurred.

2008/05/15-02:42:09, [TRCE-1001], 144,, WARNING, SilkWorm48000, Trace dump available (Slot 6)! (reason: FFDC)

2008/05/15-02:42:09, [TRCE-1004], 145,, WARNING, SilkWorm48000, Trace dump (Slot 6) was not transferred because trace auto-FTP disabled.

So it looks like something got wonky with the CP's on the 15th of May. Fortunately it's a lightly used switch in the not-yet-operational DR facility. But it's a little scary. Mostly because google doesn't come up with anything for thne 'stack trace lines. The erroshow lines come up with some vague hits, but nothing of value. A little more investigation shows that this happened on both brocade 48000 director switches. Strange stuff. Rebooting the switches fixed it but it shouldn't have happened in the first place in my opinion. Now I'm off to investigate what would have happened on the 15th of May.

Mystery volume mounted on my Macs...

I have the very nifty utility Growl. It provides "useful notifications that you control". I run several growl aware apps and an add-on utility called "HardwareGrowler" which is quite awsome. It gives you a growl notification when a device is plugged in or unplugged, for example, when I plug my network drop in, I get a notificaiton that the EN0 device became active, further it tells me that it's 100Mb, etc. I have growl set to automatically remove notices after a few seconds if the machine is not idle. But idle notices stick around until I close them.

Yesterday I had a growl notification that said "Volume Mounted Keystoen-1.0.1.340" which was strange, because a volume doesn't show up in the finder.

rwhiffen-macbook:networker rwhiffen$ df -k

Filesystem 1K-blocks Used Avail Capacity Mounted on

/dev/disk0s2 116753840 61716172 54781668 53% /

devfs 118 118 0 100% /dev

fdesc 1 1 0 100% /dev

map -hosts 0 0 0 100% /net

map auto_home 0 0 0 100% /home

/dev/disk1s2 1348 548 800 41% /Volumes/Keystone-1.0.1.340

/dev/disk2s3 97554672 60569148 36985524 62% /Volumes/external

rwhiffen-macbook:networker rwhiffen$

So now I see a new volume /Volumes/Keystone-1.0.1.340 and it has two files: Keystone.tbz install.py. And install.py has a header that says the following:



#!/usr/bin/python

# Copyright 2008 Google Inc. All rights reserved.



"""This script will install Keystone in the correct context

(system-wide or per-user). It can also uninstall Keystone. is run by

KeystoneRegistration.framework.



Example command lines for testing:

Install: install.py --install=/tmp/Keystone.tbz --root=/Users/fred

Uninstall: install.py --nuke --root=/Users/fred



Example real command lines, for user and root install and uninstall:

install.py --install Keystone.tbz

install.py --nuke

sudo install.py --install Keystone.tbz

sudo install.py --nuke



For a system-wide Keystone, the install root is "/". Run with --help

for a list of options. Use --no-processes to NOT start background

processes (e.g. launchd item).



Errors can happen if:

- we don't have write permission to install in the given root

- pieces of our install are missing



On error, we print an message on stdout and our exit status is

non-zero. On success, we print nothing and exit with a status of 0.

"""

So it seems a google product has mounted a volume with a python module in it. Funny thing is, if I google for it, I get zero hits. So I was a bit concerned about what it is exactly. If you bzip2 -d the tbz file and tar tvf the tar file it has a lot of files in it:

drwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/drwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/drwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/drwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/lrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/GoogleShared -> Versions/Current/GoogleSharedlrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Resources -> Versions/Current/Resourcesdrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/drwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/A/-rwxr-xr-x macbuild/staff 221032 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/A/GoogleShareddrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/A/Resources/-rw-r--r-- macbuild/staff 884 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/A/Resources/Info.plistlrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/GoogleShared.framework/Versions/Current -> Adrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/Keystone.framework/lrwxr-xr-x macbuild/staff 0 2008-06-04 19:36:38 GoogleSoftwareUpdate.bundle/Contents/Frameworks/Keystone.framework/Keystone -> Versions/Current/Keystone



So it seems that the Google Software Update app launched itself and downloaded some kind of update. Funny thing is, I don't recall ever telling Google it was OK for it's app to do this. So I go into Google Updater and I find out that I 'kind of did'



So I don't have the "notify me" box checked, which is why I didn't get told about it. Strange stuff. Not sure I like it.

ldap and Active directory notes...

Need a place to squirrel my notes about LDAP and Active Directory. This seems as good a place as any. Have a bunch of book marks in Del.icio.us but they're unorganized and there's no naritive to link them together.

So I'm having issues with sun's Identity sync for windows product (which they seem to have renamed). I like the product, but this issue is driving me crazy. We have an intermediate fix for now, but I need a long term fix. At any rate, here's my crib-sheet of LDAP and AD links and notes.

Finding your Active Directory Site and Domain Controllers

This is an interesting read, gives you the ldapsearch syntax to ask AD via LDAP who your domain controllers are.

Querying Active Directory with Unix LDAP tools.

Another good one, goes into some more details.

Using ldapsearch to query Active Directory

This is a Mac-slanted version of some of the same stuff covered already.

It's worth noting I'll be using the ldapsearch syntax from OpenLDAP. I used MacPorts to intsall OpenLDAP on my MacBook Pro. The Sun ldapsarch has a different syntax but can do the same things.

So one of the things I was having trouble with is finding the domain controllers so I can add the BIO domain to the ISW. I could see them, but for what ever reason ISW couldn't. I assumed that there was some issue with the username I was binding with. Turns out, that's not the case.

For the first domain I'd run the following and get:

ldapsearch -v -H ldaps://<Domain_Controller_domain1> -x -b "ou=domain controllers,DC=<DOMAIN>,DC=<FOREST>" -D "<BIND_USER>" -W "(objectclass=computer)" "distinguishedName | dNSHostName | name"

ldap_initialize( ldaps://<Domain_Controller_domain1> )

Enter LDAP Password:

filter: (objectclass=computer)

requesting: distinguishedName|dNSHostName|name

# extended LDIF

#

# LDAPv3

# base <ou=domain controllers,DC=<DOMAIN1>,DC=<FOREST>> with scope subtree

# filter: (objectclass=computer)

# requesting: distinguishedName|dNSHostName|name

#

# <DC_NAME1>, Domain Controllers, <DOMAIN1>.<FOREST>

dn: CN=<DC_NAME1>,OU=Domain Controllers,DC=<DOMAIN1>,DC=<FOREST>

# <DC_NAME2>, Domain Controllers, <DOMAIN1>.<FOREST>

dn: CN=<DC_NAME2>,OU=Domain Controllers,DC=<DOMAIN1>,DC=<FOREST>

What'd you would expect to find, the list of domain controllers. There were actually 52 in my list, but you get the idea. If you take the "distinguishedName | dNSHostName | name" off the end, it will give you a lot more details about each DC record.

ldapsearch -v -H ldaps://<Domain_Controller_DOMAIN2> -x -b "ou=domain controllers,DC=<DOMAIN2>,DC=<FOREST>" -D "<BIND_USER>" -W "(objectclass=computer)" "distinguishedName|dNSHostName|name"

ldap_initialize( ldaps://<Domain_Controller_DOMAIN2> )

Enter LDAP Password:

filter: (objectclass=computer)

requesting: distinguishedName|dNSHostName|name

# extended LDIF

#

# LDAPv3

# base <ou=domain controllers,DC=<DOMAIN2>,DC=<FOREST>> with scope subtree

# filter: (objectclass=computer)

# requesting: distinguishedName|dNSHostName|name

#

# <DC_NAME3>, BP1 Domain Controllers, Domain Controllers, <DOMAIN2>.<FOREST>

dn: CN=<DC_NAME3>,OU=BP1 Domain Controllers,OU=Domain Controllers,DC=<DOMAIN2>,DC=<FOREST>

# <DC_NAME4>, BP1 Domain Controllers, Domain Controllers, <DOMAIN2>.<FOREST>

dn: CN=<DC_NAME4>,OU=BP1 Domain Controllers,OU=Domain Controllers,DC=<DOMAIN2>,DC=<FOREST>

# <DC_NAME5>, BP2 Domain Controllers, Domain Controllers, <DOMAIN2>.<FOREST>

dn: CN= <DC_NAME5>,OU=BP2 Domain Controllers,OU=Domain Controllers,DC=<DOMAIN2>,DC=<FOREST>

Almost what you'd expect to see. And I thought it was exactly what I'd expect to see. But what I noticed later is that there's an extra OU, which adds no value, I might add, no policies are set at that OU level, inserted into the domain structure. Someone wanted to tidy up and made OU=BP2 Domain Controllers and OU=BP1 Domain Controllers and flung all the DC's into those two folders. Turns out ISW doesn't traverse sub-OU's and that's why it wasn't getting the missing DOMAIN2. Quick fix was to move a few DC's up a level, let it do the discovery and then run with it. But if I didn't have the ldapsearch ability, I probably would have never figured it out.

Further investigating has put me in the hunt for how to find out who has the PDC FMSO role via LDAP, because that seems to be where I'm falling down now. The adventure continues and I'm sure I'l be updating this...

Sun Identity Sync for Windows troubles...

Sun's Identity Sync for Windows has been a pain in my butt lately. Turns out ISW doesn't make it easy to point at new global catalogs.

Products in play:

• MS WIndows AD (unsure of AD version right now, servers are 2k3).



• SunONE identity Server 5.2 patch 2



• Sun Identity Sync for WIndows 2004Q3

So what brought this about? At ${CLIENT} they have an application that for various reasons can not authenticate directly against AD for credentials and group memberships. It needs some attributes set for the user and setting those in AD was problematic, if I remember correctly. This was set up a long time ago and a lot as changed, so if we were doing this from scratch today, it's likely that this version could be made to work with AD. The latest version of this Documentum product does support native AD anyway so it's kind of a moot point. Anyway, on the 'issue' at hand.

The theory of operation is that an account is created in AD, added to the appropriate AD ou's, groups, and given the correct attributes. ISW then takes that information and creates the corresponding LDAP entries. After creation, ISW then montiors AD for changes and makes the corrisponding LDAP changes. It does this by talking/listenting to the Domain Controller (DC) with the PDC FSMO role. Now when a user changes their password, the ISW sees this. But it's not a domain controller so it can't actually capture the password. So ISW does the next best thing, it tells LDAP that the stored password it has is invalid and that it needs to verify AD for it next time the user logs in. So if I change my password, the LDAP password entry gets set to PW-NEEDS-SYNC (i'll try to find an actual entry to get the 'real' value). Anyway, when I try to log in via LDAP, the LDAP server takes the uname and password credentials I have provided and does a bind to an AD server with my credentials. It's the actual password I provided, not an encrypted hash or what not, which is why all of this is done over SSL This uname/password pair is then used to attempt a bind to the AD servers via LDAP. If the bind is succesfull, LDAP does two things: One, it lets me into the app. Two, it stores that password it just used as the new LDAP password, and presto, it has my 'changed' password. Pretty clever.

Now on to my story. So this has been working, not perfectly, but it worked, It talked to three AD servers I'll call them DC2, DC4 and VDC1. So, DC2 is my PDC role, and DC4 and VDC1 are the DC's we were using to talk to the two domains in play. Well, $CLIENT is moving their data center from Falls Church, VA to a CSC facility in Chicago. The datacenter move as been a great project. I should write up some of my experiences with that. It went about as well as could be expected. Part of this is to migrate to new domain controllers. So my DC2, DC4 and VDC1 are going away. Three big problems: One, DC2 is the SSL-CA for the enterprise (they failed to notice this and migrate it's services). Two, the AD team didn't realize they had an ISW/LDAP dependancy. Three, ISW doesn't let you simply repoint the Global Catalogs or Domain Controllers. Anyway, water under the bridge, we stood up a new enterprise CA, installed certs on the DC's and allowed LDAPS on them.

The big troubles have been with repointing ISW. Turns out you have to uninstall the ISW connectors and reinstall them to repoint them. Once you do that you can specify a new global catalog from which the ISW can learn the PDC FSMO role from. One issue we did come across is that when the ISW does a search for DC's in the GC, it does something equivelent to:

ldapsearch -v -H ldaps://<DC-hostname> -x -b "ou=domain controllers,DC=<DOMAIN TREE STUFF>t" -D "<USER TO LOG IN AS>" -W "(objectclass=computer)"

But it doesn't traverse sub-folders. So if your domain controllers are 'organized' into sub-folders, you're sunk. This happened in my case. The quick fix was to move a few 'core' DCs to the top level. This allowed ISW to find the DC's to talk to. I'm still having issues getting it work with the GC correctly for some reason. My ticket has been open with SUN for over a month now.

More to come I'm sure...

HP's Array-based Replication Cookbook

Found the following article on the Wikibon storage portal today.

HP's Array-based Replication Cookbook

It came out in December of 2007, but it provides a great overview of how a potential oracle 11G with ASM (booooooo) and HP EVA8x00 arrays would work.

From the PDF:

This white paper provides a comprehensive set of test-proven best practices for properly configuring, deploying, and operating an Oracle 11g database with Oracle’s Automatic Storage Manager (ASM) on an HP Enterprise Virtual Array (EVA), using Continuous Access (CA) as the remote copy infrastructure.

It's a good read, even if you're not an EVA or ASM person. It's a revist of the same topic they did for 10G/ASM/8x00. Nice to see them update it, although they seem to have forgotten to update some of the graphics...

Ooops... Should be 11G RAC.

Rijndael animation...

So if you ever wanted a a graphical representation of how Rijndael works, want no more.

Formaestudio.com hosts a great rijndael inspector animation.

It walks through the encryption process step by step.

Playing with simple logo ideas...

So I need a logo. Well, not a logo so much as a graphic for my simple website. I'm playing with "Art Text" a neat app for making text into logo-esc graphics.

So far, I'm toying with:

or

or

Perhaps I'd do it in a green glowing font to look like an old-school terminal, which is kind of the effect I want. All these are in blue simply because blue is my favorite color.