For the Envoy that is. Today is the first time I have spent some serious cash on the Envoy… I woke this morning and decided today would be a good day to go up to Discount Tire and take advantage of their free tire rotation. I knew the tires were starting to round on the outside and seeing as I had never had them rotated in the almost two years I have had it, thought it was about time. I bought this Envoy used and am pretty sure these are the original tires that came with the truck when it was new, additionally I don’t know anything about the previous owners maintenance, so I have to assume they have never been rotated. After sitting at the store for about 30 minutes while other vehicles were worked on, a technician came out and called for me. As it turns out the tires were more worn than I had realized. All four tires were either down to or below the wear bar for there tread depth. As such I had to make the decision to go ahead and replace all four tires. I went with the General Grabber HTS tire based on some discussion with the technician. With the addition of the warranty package and tire disposal and such, when all was said and done I ended up spending $710. But hey, I got that free tire rotation! 🙂
History and Home
What an incredibly long day! We woke up this morning at about 8am local time and after getting dressed we went down to the lobby and had breakfast at Starbucks, Kylie was happy they had a very nice sized chocolate doughnut that she choose to have. After we finished eating I went and asked the valet to pull the Envoy around, knowing it would take them about 10 minutes to do so, and we all went back up to the room to verify everything was packed. We then proceeded to check out and head over to the Field Museum. We made it over to the Museum Campus fairly easily and when we came out of the parking structure we realized that we were right next to Soldier Field which was pretty cool, but unfortunately our schedule didn’t allow us time to look around there, maybe next time; then it was on to the museum. Oddly enough it turned out to be cheaper to become members of the museum (which will also get us passes to a few places back in MI as well) than it was to simply get day passes, so that’s what we did. Then it was exploring time! Currently the museum is running special exhibits on whales and horses in addition to all of their normal exhibits. Their traveling exhibit of SUE, “the largest, most complete and best preserved Tyrannosaurus Rex ever discovered,” was found in the dusty bad lands of South Dakota! A fact sheet on SUE can be found here (pdf). All the exhibits were awesome as far as I am concerned, but then again natural history has always been interesting to me. The whales were incredible and seeing the skeleton of one of these large creatures was almost overwhelming. We left the Field Museum, and Chicago as a whole, right about 4pm local time and headed back home, by the time we made it here it was about 11pm. Over the course of this trip we took somewhere around 900 pictures, so in the coming days I will need to find time to prepare them and get them in to the gallery. But now, it’s time to sleep.
A Day with Fish and Dolls
Today we got up early and had breakfast in the lobby of the hotel at the State Street Bread Company, which I personally feel was very over priced and for that price the food was just ok. After eating we grabbed another taxi (which, again, Kylie loved. I am starting to think the taxi rides are going to be her favorite part of this trip) and headed over to Shedd Aquarium. We got there shortly after they opened and got our Total Experience passes which let us explore the entire aquarium, attend one of the 4-D shows, and attend the live aquatic show. The live show was pretty cool and consisted of a four Pacific White-Sided dolphins. Additionally the also showed two different Red Tailed Hawks that also live at the aquarium. We all like a lot of different parts of the aquarium but some of the highlights were the dolphins, the beluga whales (both of which we could see swimming in there tanks below the water through some very large windows). The wild reef exhibit was also really cool and Kylie was really happy to be able to see her saw nosed shark swimming around. The last thing we saw was the Caribbean Reef exhibit, and we got there just in time to see the daily dive show where one of the staff actually was swimming in the tank and feeding the fish while answering questions from the audience and telling us about the fish. The large sea turtle named Nickle that was rescued after getting hit by a boat and the 7 foot moray eel that actually came out of his hole during the presentation and swam around the diver. After we decided we were done at the aquarium (which we feel we completely explored in the almost 7 hours we were there) we went picked up another taxi (and again Kylie was thrilled) and headed back over to the Magnificent Mile, specifically to the American Girl store. We were there specifically for Kylie because she wanted to get Alissa (her doll) her very own school backpack set which we found after asking two different associates (the first told us to look upstairs, after looking for a while the second showed us where it was). Additionally Kylie also got a little surprise from Daddy and Melissa when we decided to also get Alissa a new pet, Sugar, that Kylie had showed interest in. After leaving the store we walked back down Michigan Ave, both because that was the direction of our hotel and because we wanted to see more of what the Magnificent Mile had to offer. Finding dinner tonight proved difficult for multiple reasons; the restaurants along the mile were either on the expensive side or their menus didn’t seem to offer something that all of us could eat. In the end we walked all the way back to the hotel and decided to put everything from the day away and then just eat in the lobby again, this time at their River North lounge. The food there was ok as well, but nothing spectacular, however the prices I felt were more on par with the food and its quality. The one thing I did not like was that the waiter simply stood by our table and kept engaging us in conversation as if he had nothing else to do and we weren’t trying to eat. I guess this may be because for a good portion of the time we were the only ones there, but still at least let us eat. After dinner we went back up to the room to start packing things up so we will be ready to leave in the morning. We will be going from the hotel to the Field Museum and then home from there, so we have to have everything packed and ready to check out in the morning.
Chicago Bound
Today Kylie, Melissa, and I all piled in the Envoy and headed for Chicago. Out of the three of us I am the only one to never have been to Chicago. Kylie has already been there at least twice with Mommy and has even been to at least one of the places we plan to visit, Shedd Aquarium. As for Melissa, she has also been there a few times for various different reasons. None the less we are all looking forward to the rest of this trip. I say rest because we are already out here. 🙂 The drive went relatively well, we played a few games in the car to help pass the time and it wasn’t until we were already in Chicago that we ran in to a snag. We missed a turn and got off the expressway at the wrong exit and had to find our way to the hotel via my phone and Google Maps with GPS. Needless to say we found our way just fine and checked in to the hotel about 3pm local time. After getting everything in to the room and taking a minute to get settled we then decided to hop in a taxi (which Kylie loved) and headed over to Navy Pier. It was really neat to see the Pier and all the little shops that lined it. Walking through the mall of shops on the inside was nice, and the Smith Museum of Stained Glass had some amazing works in it. After walking around the Pier we decided we were all hungry, so we had dinner at Jimmy Buffett’s Margaritaville, which was a fun place with lots of music playing and the food was pretty good too. After we were done we decided to walk back to the hotel and we took a route that would take us to Michigan Ave so we could walk along part of the Magnificent Mile and see some of the sights along it. Tomorrow is going to be another long busy day.
OnTAP version issues
Last week (on the 24th) I performed a routine OnTAP upgrade across 5 of my 3170 filers; I upgraded from 7.3.5 to 7.3.5.1P5. This upgrade was performed to help prevent a system panic from happening, which had happened twice before on our stand alone 3170 snapvault target system, under Bug 446493. Here is the Bug description:
Much disk and shelf hardware can be managed by an ANSI-standard technology called SCSI Enclosure Services (SES). Â To support SES-related processing, the SES subsystem of Data ONTAP schedules various periodic actions, using an internal timeout mechanism.
Due to a software defect, under certain conditions, the SES subsystem may set an excessive number of timers, with new timers being set before old ones expire. Â If this continues, an internal callout table will fill up, triggering an interruption of processing.
One condition in which the problem can occur is during initial setup and configuration of the storage system: Â when the “cluster-setup wizard” is run, it asks the administrator for configuration input as follows:
Do you want to create a new cluster or join an existing cluster? {create, join}: (Login timeout will occur in 60 seconds)
When installing releases in which the defect is present, if this prompt is allowed to time out, an interruption may occur at some later time.
However, the callout table can also fill up during routine production, if storage events occur in rapid succession, such that SES scans are rapidly invoked. Â Such events may include:
- Â a continuing series of disk errors
- Â breakages in disk-communication links
- adding new shelves
- power-cycling shelves
- removing a power supply
- shelf firmware updates
- shelf faults
- takeover/giveback
We were hitting this bung under the “continuing series of disk errors” event, which caused the SES scans to fill the timer-callout table. When this happened the system would panic and reboot. After the upgrade I performed all standard checkouts and everything appeared to be functioning normally and within standards, so I closed the upgrade processes and marked it as successful.
Then on the 26th we attempted to perform an allocation using the NMC. While going through all the steps everything appeared as though it was going well, all of the checks passed and we hit commit, only to have the process came back with an error message indicating that the process had failed. After opening a support call with NetApp and providing both screen captures of the error received and steps to reproduce, it was determined that we had run it to another bug. This time we hit Bug 474612. Here is the bug description:
system-cli API returns cli-result-value with an invalid return status. This invalid return status may break OnCommand and other third party applications utilizing NMSDK.
Basically what is happening is the NMC is executing the requested commands, but the filer cli is returning a response code that the NMC is not expecting and in fact does not recognize. When this happens the NMC does not know how to proceed and the command fails without performing the provisioning. Apparently this issue was introduced in OnTAP 7.3.5.1P4 and still exists in 1P5, so the solution is to downgrade OnTAP to 7.3.5.1P3. In this version of OnTAP bug 446493 is resolved and bug 474612 has not yet been introduced (as of this writing, bug 474612 is NOT resolved in any version of OnTAP).
After performing the downgrade tonight to OnTAP 7.3.5.1P3 I performed all normal checkouts and additionally performed the pending allocation via the NMC to verify the functionality and non-existence of bug 474612 in this version of OnTAP. Happily the allocation went through without issue, and from what we can tell all aspects of the filers are functioning normally. Our next OnTAP upgrade will have to be to 7.3.6.
Xbox down…again
Well isn’t that just my luck, and the reason I prepare for such! The new replacement Xbox decided to stop reading disks yesterday, and today is exactly 30 days since I got it. I took the unit back to GameStop and they replaced it under the original 30 day warranty on the unit. They also gave me a new 30 day warranty on this unit, cancelled the one year warranty I purchased on the now dead unit, and reapplied it to this new unit so that I will have a a full year warranty on this unit. I hope this one will work better, and longer.
Thoughts of my Father
In five years I do not think a day has gone by that I have not thought about my Father. I see him every time I look in the mirror, and hear his voice when I need advice. He is with me everyday, in my memories and in the love I share with my daughter. I learned how to be a man by watching him. I’m not going to say he was perfect, because he wasn’t, but he did try. I learned a lot from him, both things I wanted to emulate and things I learned that I did not. He taught me many things, both tangible things like shaving and non-tangible things like integrity. There were things, too, that I could not learn from him because by the time I was old enough to learn he was to sick to be able to teach me, like hunting. But we still spoke about those things, sometimes for hours on end, and threw those conversations he was still able to pass on his knowledge to me. My father never graduated high school, and yet he was one of the smartest people I’ll ever know. I remember seeing him read a lot, and watching a lot of TV shows that, at the time, I may have thought were boring, but now realize they were very educational. Then I realize that this is probably why I also enjoy watching, for instance, the History Channel so much. I’m not sure what has made me have this reminiscence today, but I feel close to him today and I am happy. I miss you Dad, and I love you.
Peregrine Falcon Rescue
Today Kylie, Melissa and I had a very strange, and probably rare, experience. We got to help rescue an injured Peregrine Falcon. We were on our way to Target to get Kylie anew bathing suit, and as we passed the car wash next to Meijer Daddy saw the Falcon standing under a tree. After turning around and going back we were realized that the Falcon was injured. There was another lady there who said she had already called the humane society and was waiting for their emergency animal pick up, so we went on to Target. While we were there Daddy made a few phone calls to track down other possible places for the falcon to go, but none were available. When we left to back home the falcon was still there and other people had gathered as well. This time when we pulled in we found out that the emergency pick-up personnel from the humane society was not going to be able to come for the falcon as they were dealing with another issue elsewhere. They suggested that if someone could get a blanket over the falcon and gently wrap it, that we could then put the raptor in a box and bring it to the humane society. Other people were currently trying to get the falcon, but all they had was a T-shirt, and that’s when Daddy remembered that we have a blanket in the truck. After getting the blanket, and working in coordination with the two other people who were trying to get the falcon, Daddy was able to get behind the falcon and drape the blanket over it and then gentle hold and wrap it, pick it up and place it in a box. We then followed some other helpers to the humane society. When we got there we gave the falcon to their emergency unit, but they came back out a few minutes later and told us they were not going to be able to save the falcon; it’s wing was badly broken and it’s one claw was badly mangled, unfortunately that had to euthanize the falcon. Even though there was a sad ending for the falcon, we all felt really good that we at least tried to help the bird. We were able to take a few pictures of the injured falcon, also a video of us catching the falcon.
The problems with combining FC and SATA drives on the same NetApp
The idea of tiered storage is something many business are seriously exploring now a days, and combining and leveraging it with and for “cloud” operations is a major focus. The idea behind tiered storage is that you have different levels of disk that have different performance characteristics, with the main focuses being speed and performance. We recently looked in to the possibility of adding some tiering to one of the NetApp environment’s I manage. The idea was to use 300GB FC (Fibre Channel) as our Tier 1 NAS disk and some 1TB SATA as our Tier2. On the surface of things this seems like a good idea for a couple of reasons:
- The Tier 1 disk is sufficient enough to run Oracle databases over NFS if those databases are configured properly.
- The Tier 2 disk is much cheaper and would be perfect for housing non-critical non-performance intense shares, such as home directories, at a fraction of the cost
- The mix of available disk would allow us to tailor the allocations to the actual needs of the project based on their performance requirements.
So as I began to look in to this course of action I discovered a few things that completely negates this idea, at least for having it housed in one filer (or a clustered pair even):
- When using FC connectivity to disk shelves, FC drive and SATA drives must go on different loops. Â This means that if you add a shelf of SATA to an open FC port on a filer, you will not be able to add any FC drives to that loop.
- All write operations are committed to disk as a group, regardless of what the destination aggregate is for those writes. Â So, on a system with SATA and FC disk, write operations destined for FC drives may be slowed down by the writes going to the SATA drives, if the SATA drives are busy.
The first point, the dedicated loops, isn’t such a big deal if you are planning on adding a full loop worth of SATA shelves (6 shelves per loop) and you have an open FC port with nothing else attached (or can move other shelves to fill in open spots on other loops to free up an FC port). So while the dedicated loop can be resolved and may or may not be an issue depending on your set up, it’s the second point that poses the most trouble in our environment (and I would assume most others as well).
Running the risk of impacting performance to your Tier 1 disk is not acceptable. The applications running on that tier are there for a reason, they need the performance of those faster disks. But how do you know if you will hit that impact, maybe it won’t apply? Good question. Perhaps this won’t be an issue for your environment. So ask your self this: Do you know the exact details of your workloads? No of course you don’t. You may know that there are databases on some of the exports, or that certain exports are used for regular CIFS or NFS shares for home directories, but you most likely do not know all the intimate details of each given applications work load. Without that precise knowledge it is nearly impossible to quantify the potential impact ahead of time, and thus this possible latency becomes a real concern.
Because of these factors we choose (and I recommend) not mixing FC and SATA shelves on a single (or clustered) system. If you need to have multiple Tiers you still have options:
- Implement SAN as your Tier 1 storage and utilize NAS as Tier 2
- Implement a Tier 1 NAS environment and a Tier 2 Nas environment on separate hardware (read: separate physical systems, either single headed or clustered)
- Look into an appliance that can handle different types of disk in the same housing without impact and configure tiering therein.
Tiering your storage is great idea and allows for many flexibilities and possible cost savings for the customer in terms of charge backs for disk utilization. Even so, you still need keep performance in mind, and for me, the possible performance impact is not worth the risk for mixing these shelf types on a single head.
Hopefully Just a Summer Break?
After a little over a year of playing, today Kylie has informed me that she no longer wants to play violin. She called me this morning while she was on the way to Aunt Melissa’s for the day and she said she doesn’t want to play anymore and asked if that was ok. Even though it makes me very sad as I thought she was doing very well and I personally love the sound of violin and fiddle music, I have no desire to force her to do something she is not interested in, so today will be her last day. She will go tonight and talk with Aerial, her instructor, and say good bye. Mommy and I think that this may just be a set back caused by the unstructured nature of summer break and are hoping that she might pick it back up once school starts up again in the fall. We think this because it seems since school let out Kylie has been getting a little more unfocused on the violin and we think it is due to the lack of structure during the rest of her day on Monday and in the other days of the week. I guess we will just have to wait and see what happens come fall, personally I hope she picks it back up.