{"id":6042,"date":"2016-04-13T10:29:43","date_gmt":"2016-04-13T09:29:43","guid":{"rendered":"http:\/\/www.walkingrandomly.com\/?p=6042"},"modified":"2016-04-13T10:29:43","modified_gmt":"2016-04-13T09:29:43","slug":"i-asked-twitter-whats-wrong-with-the-lustre-filesystem","status":"publish","type":"post","link":"https:\/\/walkingrandomly.com\/?p=6042","title":{"rendered":"I asked twitter: What&#8217;s wrong with the Lustre filesystem?"},"content":{"rendered":"<p>I was sat in a meeting recently where people were discussing\u00a0High Performance Computing options. Someone had found some money down the back of the sofa and asked the rest of us how we&#8217;d spend it if it were ours (For the record, I wanted some big-memory nodes &#8212; 512Gb RAM minimum).<\/p>\n<p>Halfway through the meeting someone asked about the parallel filesystem in use on Sheffield&#8217;s HPC systems and I answered <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lustre_(file_system)\">Lustre<\/a> &#8212; not expecting any further comment. I don&#8217;t know much about parallel I\/O systems but I do know that all of the systems I have access to make use of Lustre. Sheffield&#8217;s <a href=\"http:\/\/docs.iceberg.shef.ac.uk\/en\/latest\/\">Iceberg<\/a>, Manchester&#8217;s <a href=\"http:\/\/ri.itservices.manchester.ac.uk\/csf\/\">Computational Shared Facility<\/a>\u00a0and the UK National supercomputer, <a href=\"https:\/\/www.archer.ac.uk\/documentation\/data-management\/lustre.php\">Archer<\/a> to name three. The <a href=\"https:\/\/en.wikipedia.org\/wiki\/Lustre_(file_system)\">Wikipedia page for Lustre<\/a> suggests that it is very widely used in the HPC community worldwide.<\/p>\n<p>I was surprised, then, when this comment was met with derision from some in the room. One person remarked &#8216;It&#8217;s a bit&#8230;um&#8230;classic..isn&#8217;t it?&#8217;, giving the impression that it was old and outdated.\u00a0There was not much love for dear old Lustre it seemed!<\/p>\n<p>There was no time in this meeting for me ask &#8216;What&#8217;s wrong with it then? Works fine for me!&#8217;so I did what I often do in times like this, I asked twitter:<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p dir=\"ltr\" lang=\"en\">Q for HPC followers. Is there anything wrong with using Lustre? Just been in a meeting where it was mocked as &#8216;classic&#8217;.<\/p>\n<p>\u2014 Mike Croucher (@walkingrandomly) <a href=\"https:\/\/twitter.com\/walkingrandomly\/status\/719893762722373632\">April 12, 2016<\/a><\/p><\/blockquote>\n<p><script src=\"\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<p>I found the responses to be very instructive so thought I would share them here. Here&#8217;s a comment from <a href=\"http:\/\/partialcoherence.org\/pages\/about.html\">Ben Waugh<\/a><\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\"><p>\n<a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> Lustre failures seem to be the main cause of downtime in my limited experience of large HPC systems.<\/p>\n<p>\u2014 Ben Waugh (@benwaughuk) <a href=\"https:\/\/twitter.com\/benwaughuk\/status\/719934064438943745\">April 12, 2016<\/a>\n<\/p><\/blockquote>\n<p><script src=\"\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<p>Not a great start! I&#8217;ve been hit by Lustre downtime too but, since I&#8217;m not a sysadmin, I haven&#8217;t got a genuine sense of how big a problem it is. It happens often enough that I&#8217;m aware of it but not so often that I feel the need to worry about it. Looking after large HPC systems is difficult and when one attempts to do do something difficult, the occasional failure is inevitable.<\/p>\n<p>It seems that configuration might play a part in stability and performance. Peter Van Heusden kicked off with <\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\">\n<p lang=\"en\" dir=\"ltr\"><a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> Depending on the configuration can lack resilience, but is still standard HPC parallel FS.<\/p>\n<p>&mdash; Peter van Heusden (@pvanheus) <a href=\"https:\/\/twitter.com\/pvanheus\/status\/719960551938465793\">April 12, 2016<\/a><\/p><\/blockquote>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n<p>Adrian Jackson , Research Architect, HPC and Parallel Programmer, is thinking along similar lines.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\">\n<p dir=\"ltr\" lang=\"en\"><a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> 1\/2 Yes, for a long time the metadata servers have not kept up, meaning you can destroy it with lots of small files<\/p>\n<p>\u2014 Adrian Jackson (@adrianjhpc) <a href=\"https:\/\/twitter.com\/adrianjhpc\/status\/719929436183904256\">April 12, 2016<\/a><\/p><\/blockquote>\n<p><script src=\"\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\"><p>\n<a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> 2\/2 Although I believe this is being fixed in a variety of ways.<\/p>\n<p>\u2014 Adrian Jackson (@adrianjhpc) <a href=\"https:\/\/twitter.com\/adrianjhpc\/status\/719929529872068608\">April 12, 2016<\/a>\n<\/p><\/blockquote>\n<p><script src=\"\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\"><p><a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> Also, the default settings often don&#8217;t give best large scale performance, but that&#8217;s a matter of configuration<\/p>\n<p>\u2014 Adrian Jackson (@adrianjhpc) <a href=\"https:\/\/twitter.com\/adrianjhpc\/status\/719929809372114944\">April 12, 2016<\/a><\/p><\/blockquote>\n<p>Adrian also followed up with an interesting <a href=\"https:\/\/www.archer.ac.uk\/documentation\/white-papers\/parallelIO\/ARCHER_wp_parallelIO.pdf\">report on Performance of Parallel IO on ARCHER<\/a>\u00a0from June 2015. Thanks Adrian!<\/p>\n<p>Glen K Lockwood is a Computational scientist specialising in I\/O platforms at <a href=\"https:\/\/twitter.com\/NERSC\">@NERSC<\/a>, the flagship computing centre for the U.S. Dept. of Energy&#8217;s Office of Science. Glen has this insight,<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\">\n<p dir=\"ltr\" lang=\"en\"><a href=\"https:\/\/twitter.com\/eoinbrazil\">@eoinbrazil<\/a> <a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> Lustre just gets trashed because everyone uses it (because of its cost). Parallel file systems all misbehave<\/p>\n<p>\u2014 Glenn K. Lockwood (@glennklockwood) <a href=\"https:\/\/twitter.com\/glennklockwood\/status\/719899984368414720\">April 12, 2016<\/a><\/p><\/blockquote>\n<p><script src=\"\/\/platform.twitter.com\/widgets.js\" async=\"\" charset=\"utf-8\"><\/script><\/p>\n<p>I like this comment because it fits with my own, much less experienced, view of the parallel I\/O world. A classic demonstration of confirmation bias perhaps but I see this thought pattern in a lot of areas. The pattern looks like this:<\/p>\n<p>It&#8217;s hard to do &#8216;thing&#8217;. Most smart people do &#8216;thing&#8217; with &#8216;foo&#8217; and, since &#8216;thing&#8217; is hard, many people have experienced problems with &#8216;foo&#8217;. Hence, people bash &#8216;foo&#8217; a lot. &#8216;foo&#8217; sucks!<\/p>\n<p>Alternatives exist of course but there may be trade-offs. Here&#8217;s Mark Basham,\u00a0software scientist\u00a0<a class=\"tweet-url twitter-atreply pretty-link\" dir=\"ltr\" href=\"https:\/\/twitter.com\/diamondlightsou\" rel=\"nofollow\" data-mentioned-user-id=\"0\"><s>@<\/s><b>diamondlightsou<\/b><\/a> in the UK.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\" data-conversation=\"none\"><p>\n<a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> we use it at Diamond for several of our file systems. GPFS is a bit better, but a lot more costly.<\/p>\n<p>\u2014 Mark Basham (@basham_mark) <a href=\"https:\/\/twitter.com\/basham_mark\/status\/719931484258312194\">April 12, 2016<\/a>\n<\/p><\/blockquote>\n<p>I&#8217;ll give the final word to Nick Holway who sums up how I feel about parallel storage from a user&#8217;s point of view.<\/p>\n<blockquote class=\"twitter-tweet\" data-lang=\"en\">\n<p lang=\"en\" dir=\"ltr\"><a href=\"https:\/\/twitter.com\/walkingrandomly\">@walkingrandomly<\/a> I like my HPC storage to be &quot;classic&quot; and also rather &quot;boring&quot;<\/p>\n<p>&mdash; Nick Holway (@nickholway) <a href=\"https:\/\/twitter.com\/nickholway\/status\/719935058820382720\">April 12, 2016<\/a><\/p><\/blockquote>\n<p><script async src=\"\/\/platform.twitter.com\/widgets.js\" charset=\"utf-8\"><\/script><\/p>\n","protected":false},"excerpt":{"rendered":"<p>I was sat in a meeting recently where people were discussing\u00a0High Performance Computing options. Someone had found some money down the back of the sofa and asked the rest of us how we&#8217;d spend it if it were ours (For the record, I wanted some big-memory nodes &#8212; 512Gb RAM minimum). Halfway through the meeting [&hellip;]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2}},"categories":[68,41],"tags":[],"class_list":["post-6042","post","type-post","status-publish","format-standard","hentry","category-hpc","category-parallel-programming"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_shortlink":"https:\/\/wp.me\/p3swhs-1zs","jetpack_sharing_enabled":true,"_links":{"self":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/6042","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=6042"}],"version-history":[{"count":15,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/6042\/revisions"}],"predecessor-version":[{"id":6057,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=\/wp\/v2\/posts\/6042\/revisions\/6057"}],"wp:attachment":[{"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=6042"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=6042"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/walkingrandomly.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=6042"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}