[[fester:hvalid_hdd]]

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revisionPrevious revision
Next revision		Previous revision
fester:hvalid_hdd [2017/06/24 15:22] – [Detaching The Storage Volume Before A Badblocks Test (Non-Destructive Method)] add content danfester:hvalid_hdd [2017/06/24 17:46] (current) – [Non-Destructive Badblocks Test Using tmux] brief instructions dan
Line 274: Line 274:
  
 ==== Importing An Unencrypted Volume After A Non-Destructive Badblocks Test ==== ==== Importing An Unencrypted Volume After A Non-Destructive Badblocks Test ====
 +
 +This is how to reattach a non-encrypted volume in the FreeNAS web GUI.
 +
 +  * Assuming you have selected the “Storage” page click on the “Import Volume” button (1).
 +  * If the volume is not encrypted then click the “No: Skip to import” radio button (2).
 +  * Now click the “OK” button (3).
 +
 +{{:fester:a35d0b344ca9df3be2a343e662e324d6.png}}
 +
 +This will take you to a second screen and step 2 of a 2 part process.
 +
 +  * In the “Volume:” drop down selection box (1) you should see your previously detached volume.
 +  * With the correct volume selected click the “OK” button (2) and the volume should be imported momentarily.
 +{{:fester:bbedc7d350b8ad3e7b6c5082e88e7477.png}}
 +
  
 ==== Importing An Encrypted Volume After A Non-Destructive Badblocks Test ==== ==== Importing An Encrypted Volume After A Non-Destructive Badblocks Test ====
 +
 +This is how to reattach an encrypted volume in the FreeNAS web GUI.
 +
 +  * Assuming you have selected the “Storage” page click on the “Import Volume” button (1).
 +  * If the volume is encrypted then click the “Yes : Decrypt disks” radio button (2).
 +  * Now click the “OK” button (3).
 +
 +{{:fester:60299ae30c75ad8d9c46acba16bbd91f.png}}
 +
 +This will take you to a second screen and step 2 of a 3 part process.
 +
 +  * Select the disks that form the volume from the “Disks:” window (1) (in Fester’s case this was all of them).
 +  * Now click the “Browse” button (2) and a window will pop up that allows you to load in your previously saved geli key (when creating encrypted volumes always make sure you save a recovery key).
 +  * Navigate to the location of your key and load it into the FreeNAS GUI. If all goes well you should see it next to the “Browse” button (Fester’s shows up as geli.key) (2).
 +  * Now type in the passphrase (which is a password you created when you made the encrypted volume), in the text box next to “Passphrase:” (3) (Fester very imaginatively used **test**  here).
 +  * Now click the “OK” button (4).
 +
 +{{:fester:6d58aed2a4a669b5b4dd58c1e0aeaf80.png}}
 +
 +The third and final screen will now appear.
 +
 +  * In the “Volume:” drop down selection box (1) you should see your previously detached volume.
 +  * With the correct volume selected click the “OK” button (2) and the volume should be imported momentarily.
 +
 +{{:fester:881159b5e5a8f5e1a93fe8e9e89323e7.png}}
 +
  
 ==== Destructive Badblocks Test Using tmux ==== ==== Destructive Badblocks Test Using tmux ====
 +
 +Start an SSH session and log in.
 +
 +Before starting tmux we need to enable the kernel geometry debug flags, so type in this command at the command prompt.
 +
 +''sysctl kern.geom.debugflags=0x10''
 +
 +(When all the Badblocks tests are done the kernel geometry debug flags must be returned to their normal state. Thankfully no additional command is necessary, just reboot the server as this setting is not persistent and cannot survive the reboot.)
 +
 +Now type the following command at the command prompt.
 +
 +''tmux''
 +
 +{{:fester:4a215d740e93e13f525077809182f5a6.png}}
 +
 +You should see a screen something like this. Notice the green band at the bottom of the screen, this is a tmux session.
 +
 +{{:fester:4dcfa5900d17534756dc73075b2f11b5.png}}
 +
 +I will not be running the Badblocks test on ada0 (the SSD drive) there is no point as already explained and this is a destructive test (the FreeNAS OS is on this drive!).
 +
 +This leaves the 8 data storage drives to check.
 +
 +This means I will need 8 sessions opened in tmux (open the number of sessions that suits your requirements).
 +
 +Let us start by renaming the current session in tmux to something more meaningful than “csh”.
 +
 +In the tmux window press the “Ctrl” and “b” keys together, release them and then press the “,” key.
 +
 +The bar at the bottom of the window should turn yellow and you can now delete the “csh” text and rename it (Fester called his “da0” after the drive that will be tested).
 +
 +{{:fester:1ca4b4377f61e21efd0955f084ee351f.png}}
 +
 +When you have typed in the new name press the “Return/Enter” key, the bar should now resort back to its original green colour and the session should be renamed.
 +
 +{{:fester:d507060531c4fe2526fdbe55d4423dda.png}}
 +
 +At this point we need to create an additional session and rename it for the next drive to be tested.
 +
 +To create a new session press the “Ctrl” and “b” keys together, release them and then press the “c” key.
 +
 +You should get something like this where “1:csh*” is the newly created session. Incidentally the asterisk just denotes the currently selected session.
 +
 +{{:fester:f385fa179c0c65a69fc266c35ee57a09.png}}
 +
 +Let us rename this session by pressing the “Ctrl” and “b” keys together, releasing them and then pressing the “,” key. Type in the new name and press the “Return/Enter” key (just as we did before, I called this one “da1” after the next drive to be tested).
 +
 +{{:fester:f9a8ba177073914e8d2bb88b7fbba6a2.png}}
 +
 +Navigation between the different sessions is achieved by pressing the “Ctrl” and “b” keys together, releasing them and then pressing the “n” key. This will take you to the next session along.
 +
 +Alternatively you can also press the “Ctrl” and “b” keys together, release them and then press the “p” key. This will take you to the previous session.
 +
 +By using the next and previous navigational keystroke combinations you can navigate through the different sessions, the asterisk signifying which session you are currently viewing.
 +
 +Using the key combinations already explained let us create the remaining sessions needed and rename each one.
 +
 +{{:fester:d5881d189bd31b0f972fdfd49c58b15d.png}}
 +
 +Now we can run the Badblocks tests from within tmux.
 +
 +Navigate to the first session (i.e. “0:da0”) and type in the following command at the prompt.
 +
 +''badblocks -ws /dev/da0''
 +
 +Fester uses a slightly different command to improve the efficiency of the tests with the WD40EFRX drives. These drives have a sector size of 4096 bytes (even though they report 512 bytes, naughty Western Digital). I also like a more verbose output from these tests so the command includes the –v switch. I include it here for informational purposes only.  In addition to improving efficiency, the "-b 4096" below is required with larger disks (//i.e.//, larger than about 4 TB).
 +
 +''badblocks –b 4096 –vws /dev/da0''
 +
 + If the command executes properly you should see something like this.
 +
 +You will see from the screen the completion progress expressed as a percentage (1) and any errors that have occurred expressed like this “(0/0/0 errors)” (2).
 +
 +There should be zero errors throughout the test. If you get even one error then you should return the disk for testing.
 +
 +{{:fester:f50e14a41e8e32abb7e8c1e62b9d8185.png}}
 +
 +Now navigate to the next session (in Fester’s case that is “1:da1”) and type this at the command prompt.
 +
 +''badblocks -ws /dev/da1''
 +
 +(Or Fester’s variation if it suits you better, but remember to change the drive name from “da0” to “da1”.)
 +
 +Repeat this process of changing session and running the Badblocks command for every drive in your system that you want to test. In Fester’s case this means running these commands while changing sessions each time.
 +
 +''badblocks -ws /dev/da2''
 +
 +''badblocks -ws /dev/da3''
 +
 +''badblocks -ws /dev/da4''
 +
 +''badblocks -ws /dev/da5''
 +
 +''badblocks -ws /dev/da6''
 +
 +''badblocks -ws /dev/da7''
 +
  
 ==== Non-Destructive Badblocks Test Using tmux ==== ==== Non-Destructive Badblocks Test Using tmux ====
 +
 +To do a non-destructive ''badblocks'' test, follow the instructions above, but replace -w with -n.  Example commands might look like:
 +
 +''badblocks -ns /dev/da0''
 +
 +or
 +
 +''badblocks -b 4096 -nsv /dev/da1''
 +
 +This test is intended to be non-destructive--once it has completed, the data on your disk should be unharmed.  Even so, I'd discourage running this on a disk with important data unless you have a good, readily-accessible backup.
 +
  
 ==== Stopping A Badblocks Test In tmux ==== ==== Stopping A Badblocks Test In tmux ====
 +
 +If for any reason you need to stop a Badblocks test then navigate to the applicable session at press the “Ctrl” and “c” keys together, then release them. This should stop the test.
 +
 +{{:fester:2729f13d558ab0f4f9a4edee5ebd986c.png}}
 +
 +==== Detaching a tmux Session ====
 +
 +Once you've started all your tests running, you can "detach" your tmux session to let the tests run while you do something else with your SSH session (or just close the SSH connection entirely).  To do this, press the "Ctrl" and "b" keys together, then press the "d" key.  This will return you to your SSH session, without the tmux session (you'll notice that the green bar isn't present on the bottom of the screen).
 +
  
 ==== Resuming A Session In tmux ==== ==== Resuming A Session In tmux ====
 +
 +Badblocks tests can take a long time (when the tests completed Fester was far from where he had started due to Continental Drift and Plate Tectonics).
 +
 +You do not need to keep the terminal open or the client computer switched on in order to keep the tmux session running.
 +
 +If you need to pack up for the night then just close the window that the sessions are running in (just don’t shut down the server). Then get your ferrets to shut down your client computer and your pigeons to knock up a suitable night cap (I find a Multiple Orgasm very agreeable before bed).
 +
 +{{:fester:159bf7d2a0b8a2cb328b5e8322a0a72e.png}}
 +
 +When you need to re-establish the connection with the tmux session/s simply start an SSH session and log in.
 +
 +Type the following command in the command prompt.
 +
 +''tmux attach''
 +
 +or simply
 +
 +''tmux a''
 +
 +{{:fester:29505b0dbee390dd1dddeb6dd009e50f.png}}
 +
 +This should return you to the tmux session(s).
 +
 +{{:fester:17d1b1f52baacdfbdc94391348c2f86f.png}}
 +
 +When the tests are complete navigate to an open session, note the results if you need to and then type the following into the command prompt.
 +
 +''exit''
 +
 +This will close that particular session in tmux.
 +
 +Do this for each session in turn until you have exited all the sessions in tmux.
 +
 +You will find that on exiting the last open session in tmux you will be returned to the standard SSH console (in Fester’s case PuTTY).
 +
 +Now reboot the server to reset the kernel geometry debug flags to their standard setting.
 +
 +That’s the Badblocks tests complete.
 +
 +In order to complete all the HDD validation tests we must now repeat the SMART long tests. As this has already been documented I won’t repeat it here. Just go back to the relevant section and repeat again.
 +
 +Once the SMART long tests have completed then it is time to collect the results.
 +
  
 ===== Getting Your Test Results ===== ===== Getting Your Test Results =====
 +
 +Getting your test results is always a tense moment.
 +
 +(I remember such an instance in the doctor’s examination room after an unforgettable trip to Bognor Regis, often referred to as “The Riviera of the South West”. Unfortunately the doctor confirmed Fester had come back with more than just fond memories, but with the liberal application of a strong antibiotic cream Fester was as good as new in a couple of weeks.)
 +
 +Here is how to get your results.
 +
 +(Do not start this section until all HDD tests have been completed.)
 +
 +Open an SSH console and log in.
 +
 +We are going to issue a command to each HDD/SDD in succession that will interrogate and retrieve the results of the tests stored in each drives memory using SMART commands.
 +
 +At the command prompt type in the following command using the name of the first drive you want to interrogate (in Fester’s case this is ada0).
 +
 +''smartctl –a /dev/ada0''
 +
 +{{:fester:34575799200aec7f192df26e22dfbfbf.png}}
 +
 +This should produce the following screen with the test results. The window displaying the information has been maximised (1) so it is easier to read.
 +
 +{{:fester:748fe2ad03dbdf87fbd109dc8ee80c58.png}}
 +
 +At this point Fester copies the information and pastes it into a text editor for ease of use.
 +
 +If you want to do this then select the text in the SSH console by clicking with the left mouse button where you want to begin, hold it down and then highlight the text you want to include.
 +
 +When you have done this press the “Ctrl” button and the “v” button together. This keystroke will copy the highlighted text into the clip board.
 +
 +Open the text editor you wish to use (Fester uses Notepad in Windows) and paste it into the text into the editor.
 +
 +You now need to repeat this process for the next drive in your system.
 +
 +At the command prompt type in the following command using the name of the next drive you want to interrogate (in Fester’s case this is da0).
 +
 +''smartctl –a /dev/da0''
 +
 +This will produce the next set of results in the SSH console. Copy and paste as before (if you want to).
 +
 +Now repeat the process for the next drive and the next until all the drives have been interrogated and their data copied and pasted.
 +
 +(In this way you will build up a list of each drives test results in a single text file that can be saved for examination later.)
 +
 +In Festers case this would mean issuing the following commands in the SSH console.
 +
 +''smartctl –a /dev/da1''
 +
 +''smartctl –a /dev/da2''
 +
 +''smartctl –a /dev/da3''
 +
 +''smartctl –a /dev/da4''
 +
 +''smartctl –a /dev/da5''
 +
 +''smartctl –a /dev/da6''
 +
 +''smartctl –a /dev/da7''
 +
 +These commands produce copious amounts of information about the drives. If you want something a little less gregarious then use this command instead (don’t forget to change the drive name each time, and note the capital -A rather than the lowercase -a).
 +
 +''smartctl –A /dev/ada0''
 +
 +This should produce a screen that looks something like this (much more compact).
 +
 +{{:fester:7a146af501c56cb6375b83159bf9b42c.png}}
 +
 +So you have now gathered your results, but they make about as much sense as a bacon butty at a bar mitzvah.
 +
 +What now?
 +
  
 ==== Making Sense of SMART Data ==== ==== Making Sense of SMART Data ====
 +
 +When looking at SMART data from a SMART storage device certain entries in the table are not important in terms of data integrity and health. They just give general information (e.g. Model, serial number, etc) and other types of information that could be useful in certain circumstances.
 +
 +Other entries are very important and should immediately ring alarms bells if certain values are present.
 +
 +In terms of HDD/SDD hardware validation these are the entries in the SMART data you need to scrutinise.
 +
 +|  \\ __ID#__ \\  |  \\ __ATTRIBUTE_NAME__ \\  |  \\ __FLAG__ \\  |  \\ __VALUE__ \\  |  \\ __WORST__ \\  |  \\ __THRESH__ \\  |  \\ __TYPE__ \\  |  \\ __UPDATED__ \\  |  \\ __WHEN_FAILED__ \\  |  \\ __RAW_VALUE__ \\  |
 +|  \\ 1 \\ |  \\ Raw_Read_Error_Rate \\ |  \\ 0x002f \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 051 \\ |  \\ Prefail \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 5 \\ |  \\ Reallocated_Sector_Ct \\ |  \\ 0x0033 \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 140 \\ |  \\ Prefail \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 7 \\ |  \\ Seek_Error_Rate \\ |  \\ 0x002e \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 10 \\ |  \\ Spin_Retry_Count \\ |  \\ 0x0032 \\ |  \\ 100 \\ |  \\ 100 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 11 \\ |  \\ Calibration_Retry_Count \\ |  \\ 0x0032 \\ |  \\ 100 \\ |  \\ 100 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 196 \\ |  \\ Reallocated_Event_Count \\ |  \\ 0x0032 \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 197 \\ |  \\ Current_Pending_Sector \\ |  \\ 0x0032 \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 198 \\ |  \\ Offline_Uncorrectable \\ |  \\ 0x0030 \\ |  \\ 100 \\ |  \\ 253 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +|  \\ 199 \\ |  \\ UDMA_CRC_Error_Count \\ |  \\ 0x0032 \\ |  \\ 200 \\ |  \\ 200 \\ |  \\ 000 \\ |  \\ Old_age \\ |  \\ Always \\ |  \\ - \\ |  \\ 0 \\ |
 +
 +If you get any value other than __zero__ in the “RAW VALUE” for these entries you should be suspicious of this drive and may need to return the device for testing depending on the manufacturer’s warranty.
 +
 +Another area you should look at is the “SMART Self-test log structure”. Here is an example. It will tell you if the drive passed its tests.  Any result other than "Completed without error" is cause for concern.
 +
 +SMART Self-test log structure revision number 1
 +
 +|  \\ __Num__ \\  |  \\ __Test_Description__ \\  |  \\ __Status__ \\  |  \\ __Remaining LifeTime(hours)__ \\  |  \\ __LifeTime(hours)__ \\  |  \\ __LBA_of_first_error__ \\  |
 +|  \\ # 1 \\ |  \\ Extended offline \\ |  \\ Completed without error \\ |  \\ 00% \\ |  \\ 503 \\ |  \\ - \\ |
 +|  \\ # 2 \\ |  \\ Conveyance offline \\ |  \\ Completed without error \\ |  \\ 00% \\ |  \\ 494 \\ |  \\ - \\ |
 +|  \\ # 3 \\ |  \\ Short offline \\ |  \\ Completed without error \\ |  \\ 00% \\ |  \\ 75 \\ |  \\ - \\ |
 +
 +(If Fester is misinformed about interpreting SMART data or has omitted something important please let me know and I will try to put it in the guide or you could replace this or any section with your own?)
 +
 +That’s the HDD/SDD validation completed. Now it is time to reinstall FreeNAS and create a basic server.
 +
 +\\
  
  
  • fester/hvalid_hdd.1498317757.txt.gz
  • Last modified: 2017/06/24 15:22
  • by dan