Archive for the 'Job' Category

Bring Your Daughter to Work Day 2016

Today was the day that Kylie had been waiting for just about a year. It’s take your daughter to work day! Last year was Kylie’s first time participating in this event and she had so much fun that she has been asking about it ever since. So last night after work I went over and picked her up so she could spend the night at my house to make the morning a bit easier. Seeing as the events started with breakfast at 9am I decided it would be best to not go into work at 6am like I normally do, so we both got to sleep in a little later than normal, which is always a good start to the day. Traffic on the way to work was a little heavier than I am accustomed to, and even Kylie commented on it, which I thought was funny. Regardless, we made it to the office with plenty of time and Kylie was amazed that since last year I have moved my desk and now have a standing desk instead of a sitting desk. She seemed shocked to find out that I literally stand around all day. Even more exciting for her though, was that I actually do have a chair, one that adjusts to a higher height to allow me to sit at my standing desk if I so choose. She had a lot of fun sitting in such a high chair. After about 15 minutes we went over to the main gathering area where breakfast was being provided and found ourselves a seat. After breakfast there was some dancing (though Daddy doesn’t dance) and general goofing around, and shortly after it was time for the kids to go off with their groups for a day full of activities. Due to a customer issue I was, unfortunately, not able to actually sit and have lunch with Kylie. This upset me greatly, in fact, I think, a lot more than it did Kylie. Thankfully she was able to sit with Nicole and Kira so she was not alone. I was able to at least quickly go over and let her know I wasn’t going to be able to eat with her and let her know how sorry I was so she wasn’t just sitting there wondering if I forgot or something. After eating Nicole brought her over to my desk so I could see her for a few minutes before she went off to her next activity. At the end of the day we met back up and she told me about all of the cool things she did. She said she really enjoyed the programming class and that the day in general was a lot of fun. I am very fortunate to work for a company that participates in these events and grateful to have a job where I can actually bring her in for the day. We are both already looking forward to next year’s bring your daughter to work day!

Bring Your Daughter to Work Day 2015

Today was the first time that Kylie and I have gotten to participate in bring your daughter to work day, and I have to say it was pretty cool. Kylie and I arrived at work a little later than I normally do, just because of the timing for when the day’s events were scheduled to start, but still early enough that I was able to show her around a bit. As always, she was very interested in my white board, so I thought she would get a huge kick out of our technical engineering room, which is a room where three of the walls are floor to ceiling whiteboards, and man was I right! Thankfully the boards were empty and the room wasn’t in use, so she had a blast drawing and writing on the walls for about fifteen minutes, and then it was time to start the day. The day began with us going to have breakfast together, which consisted of a catered buffet of french toast, pancakes, sausage, bacon, eggs, juice, and toast. During this time some of the committee members gave a small presentation outlining what the day was going to consist of and telling the kids how to find out what group they were a member of. Then she joined her group, met her group leader, and went off for a day of adventure. The day was broken down into about 7 activities that the kids rotated through. Each activity was presented by a member of our staff and the topics ranged from learning to code, to building wooden cars depending on the age range of the groups of kids. We then all went back to the meeting area and met for lunch, which was the always popular pizza. Kylie told me about some of the things she had done so far and that she was having a really fun time, and then ran off to her next set of activities. About 3:00 we met back up again for the end of the day and I got to hear all about the fun that Kylie had. Even though she had liked all of the activities and said she had a blast at all of them, her favorite part remained the “drawing room!” She is already asking if she will be able to come back next year! I’d say today was a pretty successful day, and I hope I will be able to bring her again next year!

Company Picnic

Today was the first company (Secure-24) picnic I have gone to that I was also able to bring the family too. This was a nice change for me, at my old job (Truven Health / Thomson Reuters) the company picnic was always held during the week and during normal business hours, which meant you could only go if you could get out of the office without impacting work and that it was employees only. This was a nice change indeed! Melissa and Kylie were both able to attend and there were lots of things going on. In addition to all the normal picnic foods and drinks being supplied the company also bought out an entire ice cream truck. So we were able to walk up and order anything we wanted that they had, and it stayed there all day until they were out of everything. So they literally bought out the entire truck! The kids all seemed to love this. I think Kylie most enjoyed the bouncy house, as she stayed in there pretty much all day. A number of the management team took turns in a dunk tank and certain people had a very long line trying to dunk them! We stayed for quite a few hours and had a really good time hanging out with my co-workers and friends. So far working for Secure-24 has a lot of benefits and a really great culture and even after 10 months I am still really happy I decided to accept a position here.

From Backup to Storage

One of the goals that I set for myself when I took the position with Secure-24 was to transition to the storage team within 6 months. This has been my goal because I enjoy doing storage work a lot more than I do backup and recovery. I even mentioned in my interview (for a backup engineering position) that I did not want to do backup work and that if I were to get hired this would be my goal. Over the last few months I have been halfway to my goal due to being the point person for both storage and backups for our enterprise delivery team. In fact this team has had a major project underway for one of our customers that has us implementing two new EMC VMAX arrays that I have been deeply involved with and will begin provisioning storage next week for. However, today, my full transition has been made (sort of) due to a colleague leaving the company for another opportunity. Jon was the NetApp engineer and now that he is gone the storage team will be leaning heavily on me to fill the NetApp knowledge gap, which suits me just fine! Considering another customer is lined up to bring in some new NetApp filers as well, this puts me a great position to build another set of controllers from the ground up. So officially I have transitioned to the storage team, but until they can back fill my position I will be pulling split duty between the two teams. Needless to say I am pretty happy about these circumstances, and I even beat my goal by a month!

First full day of new career

Today was my first full day at Secure-24, and it seems like it is going to be a good place to work. My hope to hit the ground running wasn’t as fruitful as I had planned though. The environment in which the company operates is pretty locked down, which I understand the need for. However, this makes for a lot of jumping through hoops in order to connect to the desired systems where work needs to happen. This means there are also many accounts that need to be configured to allow the appropriate access, and all of mine have not been set up yet. Hopefully over the next few days things will get straightened out. From what I can tell so far working here is going to be an entirely new experience from what I am used to. I already see a lot of exciting challenges and opportunities to learn new skills and improve on existing ones!

Last Day, Technically

Today was my last day at Truven and it was a lot easier than expected. This is mainly due to the fact that my manager had me not even come in to the office, but rather we all just met up for lunch at Ichiban. I have to admit, I am going to miss those lunches as Ichiban is a great place to eat, especially when it is free! In all seriousness though yesterday was a long day of saying goodbye to people I have spent the better part of seven years working with. I am sure I will keep in touch with some of them, especially Chuck, but it’s still hard to say goodbye. I am still very excited though and looking forward to the new adventure!

Time For A Change

After 7 and a half years, and what I consider a good career, it is time for a change. Two weeks from today will be last day at Truven Health Analytics. It’s been a good career as far as I am concerned and I have come a long way from where I started, however, all good things must come to an end.

It all started in February of 2005, with a bang I might add, when I started working for Thomson Medstat as a Computer Operator on the midnight shift. After about a year and a half I relocated to Eagan MN to help rebuild the company infrastructure and move our data center from Ann Arbor to Eagan. I went out on September 16, 2006 and was only supposed to be out there for about 3 months, but that turned into almost 9 months. I made the long drive home on April 10, 2007 and during that entire 12 hour trip I was driving home to no job, as I had just relocated the data center I once worked at to Eagan! However after being home for only a few hours that day I received a phone call and got some good news, and that is when I stepped away from Computer Operations and started down the road to being a Systems Manager.

In the beginning my focus was strictly on TSM administration but sometime in 2008 I started working with NetApp filers and my career as a Storage Engineer came along with it. However in typical corporate form I wouldn’t get any training for my new position until almost a year later, and an additional two years before I got certified. The past year and a half have been spent trying to hone my skills on the NetApp’s along with trying to learn how to provision SAN disk on both EMC and HDS hardware with both Brocade and Cisco fabric switches. Admittedly I am stronger on the EMC hardware (specifically the Symmetrix line) than I am on the HDS hardware, but I think that is because I have had a lot more secondhand exposure to those. Sometime in there I got raised to a Sr. Storage Engineer as well and ran point on a number of different projects including upgrading ONTap versions, performing filer head swaps, and even upgrading the TSM systems, hardware and implementing encryption.

Yet after all this it has come time to say goodbye. I have been given the opportunity to join a new company out in Southfield, Secure-24. Secure-24 is a hosting provider for other companies, supplying everything from SAP and Oracle hosting to total IT outsourcing. As for me, I’ll be joining their backup team and focusing on NetWorker with a little bit of Avamar thrown in. Though in all honesty I am hoping to quickly transition over to their storage team as I think that is really the direction I want my career to follow.

While I was not actively looking for a new job (a recruiter for the company actually contacted me via LinkedIn) too many stars aligned and showed me that it was a good time to make a change. For instance Truven is currently gearing up for a data center migration now that the company has parted ways with Thomson Reuters and become its own stand alone company. Which I guess makes the fact that I am going to work for a data center provider a little ironic. However, I also feel that I have hit a professional growth wall here. While it is true that I am still learning much of the SAN side of the house, that learning is slow at best due to the siloed nature of such a small team (there are a total of 3 engineers, 1 contractor in India, and our manager). Those coupled along with a few personal reasons have made this opportunity to hard to pass up.

So it is with both great sadness and great nervousness and expectation that I have put in my two week notice of resignation with the only company I have known in my professional career.

OnTAP version issues

Last week (on the 24th) I performed a routine OnTAP upgrade across 5 of my 3170 filers; I upgraded from 7.3.5 to This upgrade was performed to help prevent a system panic from happening, which had happened twice before on our stand alone 3170 snapvault target system, under Bug 446493. Here is the Bug description:

Much disk and shelf hardware can be managed by an ANSI-standard technology called SCSI Enclosure Services (SES).  To support SES-related processing, the SES subsystem of Data ONTAP schedules various periodic actions, using an internal timeout mechanism.

Due to a software defect, under certain conditions, the SES subsystem may set an excessive number of timers, with new timers being set before old ones expire.  If this continues, an internal callout table will fill up, triggering an interruption of processing.

One condition in which the problem can occur is during initial setup and configuration of the storage system:  when the “cluster-setup wizard” is run, it asks the administrator for configuration input as follows:

Do you want to create a new cluster or join an existing cluster? {create, join}: (Login timeout will occur in 60 seconds)

When installing releases in which the defect is present, if this prompt is allowed to time out, an interruption may occur at some later time.

However, the callout table can also fill up during routine production, if storage events occur in rapid succession, such that SES scans are rapidly invoked.  Such events may include:

  •  a continuing series of disk errors
  •  breakages in disk-communication links
  • adding new shelves
  • power-cycling shelves
  • removing a power supply
  • shelf firmware updates
  • shelf faults
  • takeover/giveback

We were hitting this bung under the “continuing series of disk errors” event, which caused the SES scans to fill the timer-callout table. When this happened the system would panic and reboot. After the upgrade I performed all standard checkouts and everything appeared to be functioning normally and within standards, so I closed the upgrade processes and marked it as successful.

Then on the 26th we attempted to perform an allocation using the NMC. While going through all the steps everything appeared as though it was going well, all of the checks passed and we hit commit, only to have the process came back with an error message indicating that the process had failed. After opening a support call with NetApp and providing both screen captures of the error received and steps to reproduce, it was determined that we had run it to another bug. This time we hit Bug 474612. Here is the bug description:

system-cli API returns cli-result-value with an invalid return status. This invalid return status may break OnCommand and other third party applications utilizing NMSDK.

Basically what is happening is the NMC is executing the requested commands, but the filer cli is returning a response code that the NMC is not expecting and in fact does not recognize. When this happens the NMC does not know how to proceed and the command fails without performing the provisioning. Apparently this issue was introduced in OnTAP and still exists in 1P5, so the solution is to downgrade OnTAP to In this version of OnTAP bug 446493 is resolved and bug 474612 has not yet been introduced (as of this writing, bug 474612 is NOT resolved in any version of OnTAP).

After performing the downgrade tonight to OnTAP I performed all normal checkouts and additionally performed the pending allocation via the NMC to verify the functionality and non-existence of bug 474612 in this version of OnTAP. Happily the allocation went through without issue, and from what we can tell all aspects of the filers are functioning normally. Our next OnTAP upgrade will have to be to 7.3.6.

The problems with combining FC and SATA drives on the same NetApp

The idea of tiered storage is something many business are seriously exploring now a days, and combining and leveraging it with and for “cloud” operations is a major focus. The idea behind tiered storage is that you have different levels of disk that have different performance characteristics, with the main focuses being speed and performance. We recently looked in to the possibility of adding some tiering to one of the NetApp environment’s I manage. The idea was to use 300GB FC (Fibre Channel) as our Tier 1 NAS disk and some 1TB SATA as our Tier2. On the surface of things this seems like a good idea for a couple of reasons:

  1. The Tier 1 disk is sufficient enough to run Oracle databases over NFS if those databases are configured properly.
  2. The Tier 2 disk is much cheaper and would be perfect for housing non-critical non-performance intense shares, such as home directories, at a fraction of the cost
  3. The mix of available disk would allow us to tailor the allocations to the actual needs of the project based on their performance requirements.

So as I began to look in to this course of action I discovered a few things that completely negates this idea, at least for having it housed in one filer (or a clustered pair even):

  • When using FC connectivity to disk shelves, FC drive and SATA drives must go on different loops.  This means that if you add a shelf of SATA to an open FC port on a filer, you will not be able to add any FC drives to that loop.
  • All write operations are committed to disk as a group, regardless of what the destination aggregate is for those writes.  So, on a system with SATA and FC disk, write operations destined for FC drives may be slowed down by the writes going to the SATA drives, if the SATA drives are busy.

The first point, the dedicated loops, isn’t such a big deal if you are planning on adding a full loop worth of SATA shelves (6 shelves per loop) and you have an open FC port with nothing else attached (or can move other shelves to fill in open spots on other loops to free up an FC port). So while the dedicated loop can be resolved and may or may not be an issue depending on your set up, it’s the second point that poses the most trouble in our environment (and I would assume most others as well).

Running the risk of impacting performance to your Tier 1 disk is not acceptable. The applications running on that tier are there for a reason, they need the performance of those faster disks. But how do you know if you will hit that impact, maybe it won’t apply? Good question. Perhaps this won’t be an issue for your environment. So ask your self this: Do you know the exact details of your workloads? No of course you don’t. You may know that there are databases on some of the exports, or that certain exports are used for regular CIFS or NFS shares for home directories, but you most likely do not know all the intimate details of each given applications work load. Without that precise knowledge it is nearly impossible to quantify the potential impact ahead of time, and thus this possible latency becomes a real concern.

Because of these factors we choose (and I recommend) not mixing FC and SATA shelves on a single (or clustered) system. If you need to have multiple Tiers you still have options:

  • Implement SAN as your Tier 1 storage and utilize NAS as Tier 2
  • Implement a Tier 1 NAS environment and a Tier 2 Nas environment on separate hardware (read: separate physical systems, either single headed or clustered)
  • Look into an appliance that can handle different types of disk in the same housing without impact and configure tiering therein.

Tiering your storage is great idea and allows for many flexibilities and possible cost savings for the customer in terms of charge backs for disk utilization. Even so, you still need keep performance in mind, and for me, the possible performance impact is not worth the risk for mixing these shelf types on a single head.

NCDA Boot Camp, Day 6

I just passed NS163 and now officially have my NCDA! It will take a couple of weeks for them to ship out my certificate but the system has been updated with my passing score so I am officially certified. Not just me though, everyone in the class passed both exams, so there are now 8 new NCDA’s in the field. Congratulations to all the other guys and to Jon Presti for being a great instructor and making sure we all had the knowledge needed to pass the exams and get certified. Now to figure out what to do with the rest of my day, because my flight doesn’t leave until tomorrow. Had I known we would be done by 2pm today, I would have scheduled for an early evening flight. Oh well, looks like I get to go explore Las Colinas a little.