Wednesday, March 18, 2020

Cloud and HPC workloads part 3

   The final reason HPc workloads are slow tui migrate to the could had to do with storage. HPC generates and processes very large quantities of data.  Most have directed capacities well above 1 petabyte.  
  There are three main factors impacting HPC and cloud storage. First is that HPC applications expect a POSIX file system which is usually implemented on block devices where file system objects point to links blocks of information that can be accessed both sequentially,  or randomly.  Many times these applications utilize the fine storage like shared memory locations and modify individual blocks within the file. RFID requires low latency onthe file system as well as organized storage patterns.   Cloud storage utilizes block storage deep under the hood but limits access to the c blocks and instead serves files add objects in there storage platform.  In effect simplifying there structure and returning them as a single stream of data. You cannot easily modify the content of an object so utilizing them as memory addresses due not really work out. 
  The second shortcoming is a limitation of Acess Control in object storage which makes it difficult to secure these large databases at a granular level on the cloud. Controlling access for individual users at the object level becomes extremely challenging in object syste scenarios.  
  The final limitation is the unpredictable performance if object based storage.  The location of the object impacts the speed of retrieval as well as the refresh rate of the object.m, making it impossible to treat the object as a shared memory space for HPC applications because there is no guarentee that the object updates get stored before the file is accessesms by neighboring processes causing some major issues in code performance. 
    There are global experts working on solutions to these and other problems relating to HPc workloads in the cloud,  but I feel we're still a few years away from seeing mainstream HPC use of cloud architectures. 

No comments: