[Fedora-nightlife-list] A larger goal for nightlife

[Bryan Che]

Hi Jeff, thanks for your thoughts.  As a side-note, I spent yesterday 
flying back home and landed to see the blog post I'd written in my hotel 
room the night before had suddenly picked up a lot of attention.  So, 
sorry for the delayed response. =p

I agree that Nightlife should be a vehicle to help advance the use of 
open tools for scientific research.  In fact, I hope that Nightlife 
impacts the way that lots of different communities can leverage grids in 
the future.  Wikia has talked with some of us at Fedora about leveraging 
this project to help with their Web crawling, and I think that would be 
a great use of this resource and in the spirit of Fedora as well.

I'm not a member of the scientific research community, so I don't have 
many answers to your questions.  But, I do interact with commercial 
entities using proprietary grids as part of my day job.  And, many of 
these enterprises would be interested in the work in this area at 
Nightlife.  From my end, I'm happy to push and organize things to keep 
moving Nightlife ahead.  If there are development tasks that need to be 
done that are generally beneficial, I can help facilitate that.  I hope 
that people with deep knowledge and connections into the various 
research communities (like you) can help steer this project with expert 
guidance.

Bryan

Jeff Spaleta wrote:
> Good Alaskan Morning,
> 
> As a Board member, since the intent of creating the Fedora Grid
> project was announced, I've been trying to get an idea of what sort of
> open community we can build around it and its concepts.  Unlike most
> services inside Fedora, the Nightlife grid is not primarily aimed at
> Fedora users and contributors, its really aimed at researchers who
> have computationally intensive projects to solve.  Nightlife turns
> Fedora users into resources for researchers.  The hard problem is
> going to be encouraging researchers who are doing scientific
> calculations, who by and large are not Fedora contributors, to place
> compatible projects on the grid.  Getting Fedora users to run clients
> will be far easier a problem to solve in comparison.
> 
> So its time for me to ask the really hard question.... what's in it
> for the researchers?  I've been asking around a little and you have a
> fundamental problem. In my very unscientific estimation, small and
> mid-scale scientifiic research numerics is dominated by proprietary
> tools such as MATLAB and IDL.  I'm talking about processes that are
> easily divided up for multi-processor calculation, like particle in a
> cell simulations, or parameter studies.  There's a lot of code hand
> built code out there, in a questionably licensed state, which was
> never really meant to be distributed or used by other people.  A lot
> of that code is not something we can run natively on a Fedora system,
> nor could we even distribute the tools.  This is a problem.
> 
> Beyond that, once you get into massively parallelized simulations,
> codes still may require proprietary libraries like NAG, or even
> proprietary compiler chains.  These are the sorts of codes which
> expect to be run on large highly parallel cluster. and use openmpi or
> some other technology to negotiate disturbingly large matrix inversion
> calculations.  However, these sorts of codes are more likely to be
> organized into an active upstream project with acceptable licensing
> for easy distribution and use by researchers.  When national labs or
> universities invest in HPC linux clusters, these are the sorts of
> codes they do it for. I'm not sure these sorts of things are easily
> griddable for NIghtlife.  And even if they are, the upstream
> developers would need to make the invest to make sure these simulation
> codebases run on Fedora.  The codebases maybe licensed openly, but
> they may require proprietary dependencies to fully function. This is a
> problem.
> 
> So where does that leave us? I think a lot of work will have to be
> done to find the first few compelling research project which would be
> compatible with Fedora Nightlife.  Even if there is a need for spare
> Fedora CPU cycles, I'm not sure the need can be met using available
> open technology without researchers doing a significantly re-tooling.
> 
> So why should researchers re-tool?  As a researcher myself, I know my
> personal answer to the question.  I'm slowly re-tooling the  analysis
> I do away from the existing trove of IDL routines to python.  I could
> have just bought another IDL license and used the available storehouse
> of IDL tools built up already over the last 10 years before I joined
> this project.  And if I were compelled to by my employeer I would be
> using the IDL.  What is going to be 'enough' enticement to compel a
> large number of researchers to start doing something similar? Or
> better yet, to compel their project or institutional management to
> re-tool away from proprietary solutions?  I'm not sure the scientific
> funding structures encourage individual researchers seek open
> solutions as strongly as they could.  Will the Nightlife grid itself
> be an important enough resource to encourage that re-tooling?  I'm not
> sure.  Its novel, and its cool.. but I'm just not sure there is a pent
> up need for spare CPU cycles so severe in the academic research
> community that it would cause a migration from proprietary tools.
> 
> But I think the Niightlife grid can be a piece of a larger Fedora
> effort to establish standard open HPC software stack that encompasses
> highly distributed community grid which Nightlife aims to be, but also
> deeply parallel, tightly networked clusters which researchers have
> access to as part of their grant funding either at national lab
> facilities or locally at home institutions.  With a fully open HPC
> framework that we can ship in Fedora and thus EPEL/RHEL, we then give
> simulation codebase projects something to target as a platform.  We
> might even be able to encourage institutional big iron to participate
> as Nightlife nodes by running the open HPC stack that Nightlife is a
> part of.
> 
> So that's the challenge. For Nightlife to be compelling as a service
> to researchers, I think 'we' have to re-invent Fedora as a platform
> for scientific HPC development which can compete with the likes of
> MATLAB and IDL...even in environments which have institutional
> site-licenses for this proprietary tools.  We have to make Fedora
> compelling for HPC generally.  How do we do that?  I have no idea.
> I'm going to be trying to find the people who do have ideas, the
> ability, and the motivation to see things changed.  I will be pointing
> HPC stakeholders to this list to have the discussion.  All I have is
> questions, questions like these:
> 
> What should Fedora provide as an open functional HPC stack?
> Could such open HPC stack be configured out of the box for common usage cases?
> What large existing scientific simulation projects are candidates for
> inclusion into Fedora?
> Are those upstream project developers interested in seeing their
> framework included in Fedora?
> Can those codebase be adapted to be run on a highly distributed grid
> such as Nightlife?
> Is it possible for institutional clusters to join the Nightlife grid
> as clients?
> Why would institutional clusters choos to join the grid?
> 
> -jef
> 
> _______________________________________________
> Fedora-nightlife-list mailing list
> Fedora-nightlife-list at redhat.com
> https://www.redhat.com/mailman/listinfo/fedora-nightlife-list