Comments on: I Have (HPC) Issues http://www.linux-mag.com/id/4110/ Open Source, Open Standards Sat, 05 Oct 2013 13:48:18 +0000 hourly 1 http://wordpress.org/?v=3.1 By: clusterman http://www.linux-mag.com/id/4110/#comment-4585 clusterman Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4585 I have issues with vendors. One of the biggest challenges we face is knowing when and how to update the firmware on all the cluster components. It seems that vendors don't seem to understand how to do this either. For example, you call in a hardware issue, the help desk tells you that you need to update firmware. We tell them we have 500 nodes all running the same revision and they are not having the issue. The help desk says they should all be updated. So there in lies the next problem. The helpdesk tells you to download this ISO and boot your server off it. Hello! We have 500 of these. We actually had to do this manually one time because the firmware on our hard drives was bad and the only way to update them was to boot a floppy and perform a dos level update. Vendors need to develop automated tools for updating clusters with one command. The grid is designed so that you can submit in one place and a job goes off and does work at all the other locations. Why don't the vendors get this? They want to sell you a cluster but they want to perform break/fix on a per system level. Very frustrating. We actually had the vendor professional services team out here to help us build our last cluster. We could not believe our eyes when the Engineer proceeded to to one system at time. It was very painful. We've since home grown a lot of our own tools but it is just ridiculous that the vendors aren't providing this functionality. Does anyone else have this issue? Maybe its just the vendor we are working with? I have issues with vendors. One of the biggest challenges we face is knowing when and how to update the firmware on all the cluster components. It seems that vendors don’t seem to understand how to do this either. For example, you call in a hardware issue, the help desk tells you that you need to update firmware. We tell them we have 500 nodes all running the same revision and they are not having the issue. The help desk says they should all be updated. So there in lies the next problem. The helpdesk tells you to download this ISO and boot your server off it. Hello! We have 500 of these. We actually had to do this manually one time because the firmware on our hard drives was bad and the only way to update them was to boot a floppy and perform a dos level update. Vendors need to develop automated tools for updating clusters with one command. The grid is designed so that you can submit in one place and a job goes off and does work at all the other locations. Why don’t the vendors get this? They want to sell you a cluster but they want to perform break/fix on a per system level. Very frustrating. We actually had the vendor professional services team out here to help us build our last cluster. We could not believe our eyes when the Engineer proceeded to to one system at time. It was very painful. We’ve since home grown a lot of our own tools but it is just ridiculous that the vendors aren’t providing this functionality. Does anyone else have this issue? Maybe its just the vendor we are working with?

]]> By: mdomsch http://www.linux-mag.com/id/4110/#comment-4586 mdomsch Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4586 Dell has a set of open source tools called 'firmware-tools' which aims to solve this generically. http://linux.dell.com/firmware-tools/ is the home page.<br /> <br /> Thanks,<br /> Matt Domsch<br /> --<br /> Linux Technology Strategist, Dell Office of the CTO<br /> linux.dell.com & www.dell.com/linux Dell has a set of open source tools called ‘firmware-tools’ which aims to solve this generically. http://linux.dell.com/firmware-tools/ is the home page.

Thanks,
Matt Domsch
–
Linux Technology Strategist, Dell Office of the CTO
linux.dell.com & http://www.dell.com/linux

]]> By: kavalerov http://www.linux-mag.com/id/4110/#comment-4587 kavalerov Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4587 A lot of it is self-inflicted. The DYI approach is wrong. Why not pay a vendor to do the low-level work for you? Can not afford? that's the problem that needs to be solved.<br /> <br /> Example: firmware updates on live system. Easy to do on IBM pSeries, always has been, and least since AIX 4.3.3. Can also do it remotely, on any number of nodes. No need to reboot, or even degrade your run level. It does it for your service processor, your hard drives, storage adapters, anything. A lot of it is self-inflicted. The DYI approach is wrong. Why not pay a vendor to do the low-level work for you? Can not afford? that’s the problem that needs to be solved.

Example: firmware updates on live system. Easy to do on IBM pSeries, always has been, and least since AIX 4.3.3. Can also do it remotely, on any number of nodes. No need to reboot, or even degrade your run level. It does it for your service processor, your hard drives, storage adapters, anything.

]]> By: nospamou http://www.linux-mag.com/id/4110/#comment-4588 nospamou Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4588 To Matt Domsch from Dell...I had the exact same issue as in the article. I had to upgrade the drive firmware on 500 servers, and they were Dell. I contacted Dell technical support and they didn't have any work around for automating this. Something for Dell to work on as everything else I've had to upgrade has worked like a charm. Maybe when the VMWare hypervisor is built into the motherboard system upgrades will become alot easier and won't require a reboot. We'll see. To Matt Domsch from Dell…I had the exact same issue as in the article. I had to upgrade the drive firmware on 500 servers, and they were Dell. I contacted Dell technical support and they didn’t have any work around for automating this. Something for Dell to work on as everything else I’ve had to upgrade has worked like a charm. Maybe when the VMWare hypervisor is built into the motherboard system upgrades will become alot easier and won’t require a reboot. We’ll see.

]]> By: tuccillo http://www.linux-mag.com/id/4110/#comment-4589 tuccillo Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4589 I am not sure what you mean when you say "Programming and optimizing for this new model is now much harder. " (refering to SMP nodes in a cluster). I have been using SMP nodes in clusters for 10 years and never had any issues. Typically, I assign as many MPI tasks to a node as cores. This is really a function performed by a batch scheduler. I also started developing the Hybrid approach (MPI/OpenMP) many years ago for those instances where combining MPI and OpenMP provided a performance advantage over pure MPI (using the same number of core). I have seen very few examples, however, where this is true. Most of the time, however, I assign as many MPI tasks to a node as core. The message passing model doesnt care where the processes are located from a functionality point of view. In a very few instances there was a performance impact but arbitrary assignment of MPI tasks to arbitrary nodes is a feature of some batch schedulers. For the vast majority of MPI programs I have seen, the number of cores on a node is just not an issue. I am not sure what you mean when you say “Programming and optimizing for this new model is now much harder. ” (refering to SMP nodes in a cluster). I have been using SMP nodes in clusters for 10 years and never had any issues. Typically, I assign as many MPI tasks to a node as cores. This is really a function performed by a batch scheduler. I also started developing the Hybrid approach (MPI/OpenMP) many years ago for those instances where combining MPI and OpenMP provided a performance advantage over pure MPI (using the same number of core). I have seen very few examples, however, where this is true. Most of the time, however, I assign as many MPI tasks to a node as core. The message passing model doesnt care where the processes are located from a functionality point of view. In a very few instances there was a performance impact but arbitrary assignment of MPI tasks to arbitrary nodes is a feature of some batch schedulers. For the vast majority of MPI programs I have seen, the number of cores on a node is just not an issue.

]]> By: jmcculloch http://www.linux-mag.com/id/4110/#comment-4590 jmcculloch Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/4110/#comment-4590 Find another vendor. We employ a PXE-bootable DOS image to apply firmware updates and CMOS configuration changes on a large scale.<br /> <br /> Another option is to use a package like IBM Director with Remote Deployment Manager add-on. It works with most server vendors.<br /> <br /> Furthermore, if your cluster management package supports parallel shell, often you can copy the firmware update package locally to each system and execute at the command prompt. This usually requires rebooting the cluster to apply the update.<br /> <br /> Regards,<br /> John McCulloch<br /> PCPC Direct Ltd.<br /> http://www.pcpcdirect.com Find another vendor. We employ a PXE-bootable DOS image to apply firmware updates and CMOS configuration changes on a large scale.

Another option is to use a package like IBM Director with Remote Deployment Manager add-on. It works with most server vendors.

Furthermore, if your cluster management package supports parallel shell, often you can copy the firmware update package locally to each system and execute at the command prompt. This usually requires rebooting the cluster to apply the update.

Regards,
John McCulloch
PCPC Direct Ltd.
http://www.pcpcdirect.com

]]>