GSI Forum
GSI Helmholtzzentrum für Schwerionenforschung

Home » PANDA » PandaRoot » General » Data loss on Kronos
Data loss on Kronos [message #19785] Fri, 28 October 2016 14:14
Jennifer Pütz is currently offline  Jennifer Pütz
Messages: 47
Registered: April 2015
Location: FZ Juelich
continuous participant
From: *ikp.kfa-juelich.de
Hi all,

I am not sure if most of you received the e-mails from HPC-info@gsi.de mailing list.
There was a crash of the file system on Kronos last week. Some of the data are lost.
There is a list which contains information about lost files.

----------------------------------------------------------------
Subject: Overview of unlinked files from OST31 and OST32
Date: Thu, 27 Oct 2016 11:16:55 +0200
From: Thomas Stibor <t.stibor@gsi.de>
Reply-To: hpc@GSI.DE
To: HPC-INFO@gsi.de


To whom it may concern:

Due to an unrecoverable ZFS/Lustre crash, data stored on OST31 and OST32
are lost. The meta data still existed as "???????? ?..." filenames in the
Lustre mount point, however, the corresponding bulk data which was
stored on the OST31 and OST32 was damaged due to the crash and thus
lost. A cleanup process was started which removed the meta data of OST31
and OST32. A list of lost files is provided at:

/lustre/nyx/unlink_files_OST31_OST32/by_directories# ll
total 184590
-rw-r----- 1 root alice 121584012 Oct 27 10:28 unlink_files_nyx-directory-alice
-rw-r----- 1 root astrum 125315 Oct 27 10:28 unlink_files_nyx-directory-astrum
-rw-r----- 1 root bhs 10267950 Oct 27 10:28 unlink_files_nyx-directory-bhs
-rw-r----- 1 root bio 474985 Oct 27 10:28 unlink_files_nyx-directory-bio
-rw-r----- 1 root cbm 5978543 Oct 27 10:28 unlink_files_nyx-directory-cbm
-rw-r----- 1 root kp1 83708 Oct 27 10:28 unlink_files_nyx-directory-ceres
-rw-r----- 1 root kr 10484 Oct 27 10:28 unlink_files_nyx-directory-emmi
-rw-r----- 1 root fairgsi 49825 Oct 27 10:28 unlink_files_nyx-directory-fairgsi
-rw-r----- 1 root fn 601010 Oct 27 10:28 unlink_files_nyx-directory-fn
-rw-r----- 1 root fopi 2681846 Oct 27 10:28 unlink_files_nyx-directory-fopi
-rw-r----- 1 root ks 200937 Oct 27 10:28 unlink_files_nyx-directory-gamma
-rw-r----- 1 root had1 645167 Oct 27 10:28 unlink_files_nyx-directory-had1
-rw-r----- 1 root hades 6719671 Oct 27 10:28 unlink_files_nyx-directory-hades
-rw-r----- 1 root hht 66155 Oct 27 10:28 unlink_files_nyx-directory-hht
-rw-r----- 1 root hij 49379 Oct 27 10:28 unlink_files_nyx-directory-hij
-rw-r----- 1 root him 13402 Oct 27 10:28 unlink_files_nyx-directory-him
-rw-r----- 1 root hpc 607543 Oct 27 10:28 unlink_files_nyx-directory-hpc
-rw-r----- 1 root htit 17593 Oct 27 10:28 unlink_files_nyx-directory-htit
-rw-r----- 1 root uf7 1995904 Oct 27 10:28 unlink_files_nyx-directory-hyihp
-rw-r----- 1 root hyphi 16705 Oct 27 10:28 unlink_files_nyx-directory-hyphi
-rw-r----- 1 root land 629355 Oct 27 10:28 unlink_files_nyx-directory-land
-rw-r----- 1 root fltc 356033 Oct 27 10:28 unlink_files_nyx-directory-lcsc
-rw-r----- 1 root ul 1527490 Oct 27 10:28 unlink_files_nyx-directory-lobi
-rw-r----- 1 root root 2821922 Oct 27 10:28 unlink_files_nyx-directory-mdt1
-rw-r----- 1 root nustar 29098 Oct 27 10:28 unlink_files_nyx-directory-nustar
-rw-r----- 1 root pbar 4949855 Oct 27 10:28 unlink_files_nyx-directory-panda
-rw-r----- 1 root pbar 85 Oct 27 10:28 unlink_files_nyx-directory-pandase
-rw-r----- 1 root psl 57265 Oct 27 10:28 unlink_files_nyx-directory-psl
-rw-r----- 1 root radprot 256776 Oct 27 10:28 unlink_files_nyx-directory-radprot
-rw-r----- 1 root rz 448363 Oct 27 10:28 unlink_files_nyx-directory-rz
-rw-r----- 1 root kc 10672 Oct 27 10:28 unlink_files_nyx-directory-ship
-rw-r----- 1 root tasca 16974 Oct 27 10:28 unlink_files_nyx-directory-tasca
-rw-r----- 1 root the 15176649 Oct 27 10:28 unlink_files_nyx-directory-theory
-rw-r----- 1 root uf7 7818541 Oct 27 10:28 unlink_files_nyx-directory-uf7
-rw-r----- 1 root ukt 111191 Oct 27 10:28 unlink_files_nyx-directory-ukt

For each directory in /lustre/nyx/$DIR a file named unlink_files_nyx-directory-$DIR
is created which contains the lost files. Access is granted with proper
group read rights.

Best wishes,
Thomas
------------------------------------------------------------------------ ------------

Best regards,
Jenny
 
Read Message
Previous Topic: Setting target type in master macros
Next Topic: HGS-HIRe SUMMER STUDENT PROGRAM at G S I / F A I R, July 24 - Sep. 14, 2017
Goto Forum:
  


Current Time: Fri Mar 29 00:20:30 CET 2024

Total time taken to generate the page: 0.01012 seconds