Mdthbs

Instead of cp -a /foo /bar you can also use rsync and limit the bandwidth as you need.

From the rsync's manual:

--bwlimit=KBPS

limit I/O bandwidth; KBytes per second

So, the actuall command, also showing the progress, would look like this:

rsync -av --bwlimit=100 --progress /foo /bar

I would assume you are trying not to disrupt other activity. Recent versions of Linux include ionice which does allow you to control the scheduling of IO.

Besides allowing various priorities, there is an additional option to limit IO to times when the disk is otherwise idle. The command man ionice will display the documentation.

Try copying the file using a command like:

ionice -c 3 cp largefile /new/directory

If the two directories are on the same device you may find linking the file does what you want. If you are copying for backup purposes, do not use this option. ln is extremely fast as the file itself does not get copied. Try:

ln largefile /new/directory

Or if you just want to access it from a directory on a different device try:

ln -s largefile /new/directory

If the ionice solution is not enough (whyever) and you really want to limit I/O to an absolute value there are several possibilities:

1. the probably easiest: ssh. It has a built-in bandwidth limit. You would use e.g. tar (instead of cp) or scp (if that's good enough; I don't know how it handles symlinks and hard links) or rsync. These commands can pipe their data over ssh. In case of tar you write to /dev/stdout (or -) and pipe that into the ssh client which executes another tar on the "remote" side.




2. elegant but not in the vanilla kernel (AFAIK): The device mapper target ioband. This, of course, works only if you can umount either the source or target volume.




3. some self-written fun: grep "^write_bytes: " /proc/$PID/io gives you the amount of data a process has written. You could write a script which starts cp in the background, sleeps for e.g. 1/10th second, stops the background cp process (kill -STOP $PID), checks the amount which has been written (and read? about the same value in this case), calculates for how long cp must pause in order to take the average transfer rate down to the intended value, sleeps for that time, wakes up cp (kill -CONT $PID), and so on.

Your problem is probably not with your computer, per se, it's probably fine. But that USB flash transition layer has a processor of its own that has to map out all of your writes to compensate for what could be as much as a 90% faulty flash chip, who knows? You flood it then you flood your buffers then you flood the whole bus, then you're stuck, man - after all, that's where all your stuff is. It may sound counter-intuitive but what you really need is blocking I/O - you need to let the FTL set the pace and then just keep up.

(On hacking FTL microcontrollers: http://www.bunniestudios.com/blog/?p=3554)

All of the above answers should work so this is more a "me too!" than anything else: I've totally been there, man. I solved my own issues with rsync's --bwlimit arg (2.5mbs seemed to be the sweet spot for a single, error-free run - anything more and I'd wind up with write-protect errors). rsync was especially suited to my purpose because I was working with entire filesystems - so there were a lot of files - and simply running rsync a second time would fix all of the first run's problems (which was necessary when I'd get impatient and try to ramp past 2.5mbs).

Still, I guess that's not quite as practical for a single file. In your case you could just pipe to dd set to raw-write - you can handle any input that way, but only one target file at a time (though that single file could be an entire block device, of course).

## OBTAIN OPTIMAL IO VALUE FOR TARGET HOST DEV ##

## IT'S IMPORTANT THAT YOUR "bs" VALUE IS A MULTIPLE ##

## OF YOUR TARGET DEV'S SECTOR SIZE (USUALLY 512b) ##

% bs=$(blockdev --getoptio /local/target/dev)



## START LISTENING; PIPE OUT ON INPUT ##

% nc -l -p $PORT | lz4 | 

## PIPE THROUGH DECOMPRESSOR TO DD ## 

>    dd bs=$bs of=/mnt/local/target.file 

## AND BE SURE DD'S FLAGS DECLARE RAW IO ##

>        conv=fsync oflag=direct,sync,nocache



## OUR RECEIVER'S WAITING; DIAL REMOTE TO BEGIN ##

% ssh user@remote.host <<-REMOTECMD

## JUST REVERSED; NO RAW IO FLAGS NEEDED HERE, THOUGH ## 

>    dd if=/remote/source.file bs=$bs |

>    lz4 -9 | nc local.target.domain $PORT

> REMOTECMD  

You might find netcat to be a little faster than ssh for the data transport if you give it a shot. Anyway, the other ideas were already taken, so why not?

[EDIT]: I noticed the mentions of lftp, scp, and ssh in the other post and thought we were talking about a remote copy. Local's a lot easier:

% bs=$(blockdev --getoptio /local/target/dev)

% dd if=/src/fi.le bs=$bs iflag=fullblock of=/tgt/fi.le 

>    conv=fsync oflag=direct,sync,nocache

[EDIT2]: Credit where it's due: just noticed ptman beat me to this by like five hours in the comments.

Definitely you could tune $bs for performance here with a multiplier - but some filesystems might require it to be a multiple of the target fs's sectorsize so keep that in mind.

The problem is that the copy is filling up your memory with blocks "in flight," crowding out "useful" data. A known (and very hard to fix) bug in the Linux kernel handling of I/O to slow devices (USB in this case).

Perhaps you can try to parcel out the copying, e.g. by a script like the following (proof-of-concept sketch, totally untested!):

while true do

  dd if=infile of=outfile bs=4096 count=... seek=... skip=...

  sleep 5

done

adjusting seek and skip by count each round. Need to tune count so it doesn't fill up (too much) memory, and 5 to allow it to drain.

Lower the dirty page limit. The default limit is insane.

Create /etc/sysctl.d/99-sysctl.conf with:

vm.dirty_background_ratio = 3

vm.dirty_ratio = 10

Then run sysctl -p or reboot.

What's happening is that data is being read faster than it can be written to the destination disk. When linux copies files, what it does is read them into RAM, then mark the pages as dirty for writing to the destination. Dirty pages cannot be swapped out. So if the source disk is faster than the destination disk and you're copying more data than you have free RAM, the copy operation will eat up all available RAM (or at least whatever the dirty page limit is, which could be more than the available RAM) and cause starvation as the dirty pages cannot be swapped out and clean pages get used and marked dirty as they are freed.

Note that his will not completely solve the problem...what linux really needs is some way to arbitrate creation of dirty pages so a large transfer taking place does not eat up all available RAM/all allowed dirty pages.

This problem has nothing to do with errors or faults in hardware or software, it's just your kernel trying to be nice to you and give your prompt back and copy in the background (it uses an in-kernel cache: more RAM, more cache, but you can limit it by writing somewhere in /proc - not reccommended though). Flash drives are too slow and while the kernel writes on it, other IO operations can't be performed fast enough. ionice mentioned a several times in other answers is ok. But have you tried just mounting the drive with -o sync to avoid OS buffering? It's probably the simplest solution out there.