Mdthbs

Did you tried to examin what programs like iotop is showing? It will tell you exacly what kind of process is currently writing to the disk.

example output:

Total DISK READ: 0.00 B/s | Total DISK WRITE: 0.00 B/s

  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND

    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init

    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]

    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]

    6 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/0]

    7 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [watchdog/0]

    8 rt/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [migration/1]

 1033 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [flush-8:0]

   10 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/1]

You can enable IO debugging via echo 1 > /proc/sys/vm/block_dump and then watch the debugging messages in /var/log/syslog. This has the advantage of obtaining some type of log file with past activities whereas iotop only shows the current activity.

Assuming that the disk noises are due to a process causing a write and not to some disk spindown problem, you can use the audit subsystem (install the auditd package). Put a watch on the sync calls and its friends:

auditctl -S sync -S fsync -S fdatasync -a exit,always

Watch the logs in /var/log/audit/audit.log. Be careful not to do this if the audit logs themselves are flushed! Check in /etc/auditd.conf that the flush option is set to none.

If files are being flushed often, a likely culprit is the system logs. For example, if you log failed incoming connection attempts and someone is probing your machine, that will generate a lot of entries; this can cause a disk to emit machine gun-style noises. With the basic log daemon sysklogd, check /etc/syslog.conf: if a log file name is not be preceded by -, then that log is flushed to disk after each write.

It might be your drives automatically spinning down, lots of consumer-grade drives do that these days. Unfortunately on even a lightly loaded system, this results in the drives constantly spinning down and then spinning up again, especially if you're running hddtemp or similar to monitor the drive temperature (most drives stupidly don't let you query the SMART temperature value without spinning up the drive - cretinous!).

This is not only annoying, it can wear out the drives faster as many drives have only a limited number of park cycles. e.g. see https://bugs.launchpad.net/ubuntu/+source/hdparm/+bug/952556 for a description of the problem.

I disable idle-spindown on all my drives with the following bit of shell code. you could put it in an /etc/rc.boot script, or in /etc/rc.local or similar.

for disk in /dev/sd? ; do

  /sbin/hdparm -q -S 0 "$disk"

done

I just found that s.m.a.r.t was causing an external USB disk to spin up again and again on my raspberry pi. Although SMART is generally a good thing, I decided to disable it again and since then it seems that unwanted disk activity has stopped

You can wack on this a bit. Should narrow it down for most.

find / -mount -newer /proc -print

Give files modified since boot on the physical device of the / files system. Knowing the files will likely help identify the writer.

In case you need to narrow it down to an exact disk use the following:

run lsblk and look up the device number. In the case below it is 9:126

NAME        MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT

sda           8:0    0   7.3T  0 disk  

└─md126       9:126  0  13.8T  0 raid0 /mnt/InternalPhase

sdb           8:16   0   7.3T  0 disk  

└─md126       9:126  0  13.8T  0 raid0 /mnt/InternalPhase

sdc           8:32   0   7.3T  0 disk  

└─sdc1        8:33   0   7.3T  0 part  /mnt/InternalFBE

run lsof | grep '9,126' with the : replace with , compared to the above disk number. In my case this shows up as:

bash      389162            root  cwd       DIR              9,126      4096  449183796 /mnt/InternalPhase/0000000001/CHANNEL01/LIVE/PHASE/DATA/2018/10/04

with the PID of 389162 kill this process using:

kill -9 389162

The problem is that the system needs to flush data from the disk buffers to the disk ever 5 seconds or so by default. Thus if the disk does spin down, there will be little option other than to spin back up again when a flush needs to happen. So the problem is not really avoidable other than by disabling spins downs or disk power management features altogether hdparm -B 255 /dev/hdax. This is probably the better option since restarting so often can definitely be more damaging than simply staying on all the time.