Mdthbs

How long do you think advanced cybernetic implants would plausibly last?

Duplicate instruments in unison in an orchestra

Is gzip atomic?

Macro inserted via everypar in obeylines context doesn't see some commands

What does "rel" in `mathrel` and `stackrel` stands for?

What names do Cormyr's people use?

How were medieval castles built in swamps or marshes without draining them?

Discussing work with supervisor in an invited dinner with his family

Limitations with dynamical systems vs. PDEs?

When calculating a force, why do I get different result when I try to calculate via torque vs via sum of forces at an axis?

Why doesn't 'd /= d' throw a division by zero exception?

How does encoder decoder network works?

What is this artifact and how to avoid it?

Hangman game in Python - need feedback on the quality of code

What happened to the HDEV ISS Experiment? Is it over?

Are game port joystick buttons ever more than plain switches? Is this one just faulty?

How to check whether a sublist exist in a huge database lists in a fast way?

What is the difference between "Grippe" and "Männergrippe"?

Why are non-collision-resistant hash functions considered insecure for signing self-generated information

Why is proof-of-work required in Bitcoin?

Where can/should I, as a high schooler, publish a paper regarding the derivation of a formula?

Is first Ubuntu user root?

Higman's lemma and a manuscript of Erdős and Rado

Who was the most successful German spy against Great Britain in WWII, from the contemporary German perspective?

Why does `/dev/shm` on an EC2 instance have I/O throughput comparable to an EBS drive?

How should I set up additional EBS storage on an Amazon EC2 instance?Mount and copy /var to EC2 instance store during first bootWill swap file engage automatically when I write too much to /dev/shm

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

3

I recently learned about the convenient "shared memory" file system at /dev/shm. I wanted to see if I could use this to speed up a sometimes-disk-bound program that writes to and reads from a directory, so I ran a little experiment on an EC2 instance (c4.8xlarge running Ubuntu 16.04):

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m21.381s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m20.266s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m14.334s

The EC2 instance seems to have a write throughput to /dev/shm/, that is comparable to an EBS drive, which was surprising. htop indicates that the machine isn't using swap space in order to write to /dev/shm. The noticeably faster write to /dev/null in the third case indicates that I'm probably not bound by some other factor (e.g. CPU via the implementation of yes) in the first two cases.

I ran the same experiment on my personal computer -- enough memory that /dev/shm can hold 7.5 GB of asdfjkln by default, I can dig up more hardware details if anyone thinks they'll matter -- also running Ubuntu 16.04:

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m36.520s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m12.516s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m11.252s

This is much closer to what I expected. Read throughput (writing to /dev/null) on both machines and from both file systems is roughly proportional to write throughput in the corresponding case.

Two other observations, which I don't know quite how to interpret:

• On the EC2 instance, htop indicates memory usage comparable to the size of /dev/shm/fake.txt once it's written, while on my desktop, it does not.


• On the EC2 instance, concurrent disk write congestion appears to slow down a write to shared memory by an amount comparable to how much the writes to disk were slowed down, while on my desktop, it does not.

amazon-ec2 shared-memory

share|improve this question

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

• 
  
  
  

  
  2
  

  
  
  
  
  
  

  
  
  You really need to ask Amazon. They won't tell you, but probably nobody else can.
  
  – Michael Homer
  Jan 24 '18 at 5:36
  

  
  
  
  


• 
  
  
  

  
  2
  

  
  
  
  
  
  

  
  
  You would improve the question by adding details of the swap usage for both systems to it.
  
  – JdeBP
  Jan 24 '18 at 6:29
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  /dev/zero might (?) be a better data source. Interesting question.
  
  – Michael - sqlbot
  Jan 24 '18 at 13:11
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JdeBP neither machine is using swap, clarified in the question. Thanks
  
  – James
  Jan 24 '18 at 13:40
  
  
  

  
  
  
  

add a comment | 

3

I recently learned about the convenient "shared memory" file system at /dev/shm. I wanted to see if I could use this to speed up a sometimes-disk-bound program that writes to and reads from a directory, so I ran a little experiment on an EC2 instance (c4.8xlarge running Ubuntu 16.04):

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m21.381s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m20.266s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m14.334s

The EC2 instance seems to have a write throughput to /dev/shm/, that is comparable to an EBS drive, which was surprising. htop indicates that the machine isn't using swap space in order to write to /dev/shm. The noticeably faster write to /dev/null in the third case indicates that I'm probably not bound by some other factor (e.g. CPU via the implementation of yes) in the first two cases.

I ran the same experiment on my personal computer -- enough memory that /dev/shm can hold 7.5 GB of asdfjkln by default, I can dig up more hardware details if anyone thinks they'll matter -- also running Ubuntu 16.04:

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m36.520s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m12.516s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m11.252s

This is much closer to what I expected. Read throughput (writing to /dev/null) on both machines and from both file systems is roughly proportional to write throughput in the corresponding case.

Two other observations, which I don't know quite how to interpret:

• On the EC2 instance, htop indicates memory usage comparable to the size of /dev/shm/fake.txt once it's written, while on my desktop, it does not.


• On the EC2 instance, concurrent disk write congestion appears to slow down a write to shared memory by an amount comparable to how much the writes to disk were slowed down, while on my desktop, it does not.

amazon-ec2 shared-memory

share|improve this question

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

• 
  
  
  

  
  2
  

  
  
  
  
  
  

  
  
  You really need to ask Amazon. They won't tell you, but probably nobody else can.
  
  – Michael Homer
  Jan 24 '18 at 5:36
  

  
  
  
  


• 
  
  
  

  
  2
  

  
  
  
  
  
  

  
  
  You would improve the question by adding details of the swap usage for both systems to it.
  
  – JdeBP
  Jan 24 '18 at 6:29
  

  
  
  
  


• 
  
  
  

  
  1
  

  
  
  
  
  
  

  
  
  /dev/zero might (?) be a better data source. Interesting question.
  
  – Michael - sqlbot
  Jan 24 '18 at 13:11
  
  
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  @JdeBP neither machine is using swap, clarified in the question. Thanks
  
  – James
  Jan 24 '18 at 13:40
  
  
  

  
  
  
  

add a comment | 

I recently learned about the convenient "shared memory" file system at /dev/shm. I wanted to see if I could use this to speed up a sometimes-disk-bound program that writes to and reads from a directory, so I ran a little experiment on an EC2 instance (c4.8xlarge running Ubuntu 16.04):

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m21.381s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m20.266s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m14.334s

The EC2 instance seems to have a write throughput to /dev/shm/, that is comparable to an EBS drive, which was surprising. htop indicates that the machine isn't using swap space in order to write to /dev/shm. The noticeably faster write to /dev/null in the third case indicates that I'm probably not bound by some other factor (e.g. CPU via the implementation of yes) in the first two cases.

I ran the same experiment on my personal computer -- enough memory that /dev/shm can hold 7.5 GB of asdfjkln by default, I can dig up more hardware details if anyone thinks they'll matter -- also running Ubuntu 16.04:

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m36.520s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m12.516s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m11.252s

This is much closer to what I expected. Read throughput (writing to /dev/null) on both machines and from both file systems is roughly proportional to write throughput in the corresponding case.

Two other observations, which I don't know quite how to interpret:

• On the EC2 instance, htop indicates memory usage comparable to the size of /dev/shm/fake.txt once it's written, while on my desktop, it does not.


• On the EC2 instance, concurrent disk write congestion appears to slow down a write to shared memory by an amount comparable to how much the writes to disk were slowed down, while on my desktop, it does not.

amazon-ec2 shared-memory

share|improve this question

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m21.381s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m20.266s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m14.334s

$ time yes asdfjkl | head -1000000000 > /mnt/fake.txt

real    0m36.520s

$ time yes asdfjkl | head -1000000000 > /dev/shm/fake.txt

real    0m12.516s

$ time yes asdfjkl | head -1000000000 > /dev/null

real    0m11.252s

share|improve this question

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

share|improve this question

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

edited 6 hours ago

ctrl-alt-delor

14.1k55 gold badges3333 silver badges6464 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges

asked Jan 24 '18 at 4:43

JamesJames

1633 bronze badges