Mdthbs

What does "see" in "the Holy See" mean?

AC contactor 1 pole or 2?

401(k) investment after being fired. Do I own it?

Is my employer paying me fairly? Going from 1099 to W2

How do I stop my characters falling in love?

What is the lowest-speed bogey a jet fighter can intercept/escort?

3D Statue Park: U shapes

kids pooling money for Lego League and taxes

Is this photo showing a woman posing in the nude before teenagers real?

Is it legal for private citizens to "impound" e-scooters?

Will any serial mouse connect to Classic Macs?

Why can't my huge trees be chopped down?

How can I prevent corporations from growing their own workforce?

High income, sudden windfall

Why did Saturn V not head straight to the moon?

How to change the font style (not the size but the style) of algorithimc package

How do I address my Catering staff subordinate seen eating from a chafing dish before the customers?

Heisenberg uncertainty principle in daily life

Send a single HTML email from Thunderbird, overriding the default "plain text" setting

How to start an application when a specific disk is mounted

Is it legal to use cash pulled from a credit card to pay the monthly payment on that credit card?

Giant space birds hatching out of planets; short story

Are there any examples of technologies have been lost over time?

What is the difference between 1/3, 1/2, and full casters?

How can `cat /proc/$pid/cmdline` take several seconds?

Suggestions needed to debug why ps -ef gets stuckHow does /proc/$PID/fd actually work?Why does a process of a binary with only execute permission remain hidden in “ps” when using hidepid=2, if the user is not root?What is the BOOT_IMAGE parameter in /proc/cmdlineEncoding of /proc/<pid>/cmdline filesDuration of /proc/pid/statsudo pgrep -f matches arbitrary strings and returns increasing pidsCan /proc/PID still exist when PID isn't running?Unexpected non-null encoding of /proc/<pid>/cmdlineWhy is cat /proc/[pid]/pagemap not giving any output?Why is the sum of /proc/pid/stat CPU greater than /proc/stat

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

4

1

I encountered this strange behavior yesterday on one of our servers. ps, pgrep and htop (on startup) were very slow. strace ps showed that read('/proc/$pid/cmdline) took several seconds on some processes. Why did this happen?

Some observations:

• The processes executable was on NFS


• The processes (about 20+) were doing unlink and symlink operations on files also on NFS, in parallel


• They're forked from the same parent process


• There're 80GB of RAM available (mostly cached), but swap (only 4GB) is in full use


• I run while true; do cat /proc/$pid/status; sleep .1; done, cat returned immediately if State is S or R, but took several seconds when State is D
  

I did some Google'ing and found some SO answers suggesting that when State is D, reading /proc/$pid/cmdline would stall. Is that true? And how does that work? Why was /proc/$pid/cmdline, which was set before the program started, affected by what it was doing after that?

• https://unix.stackexchange.com/a/109864


• https://superuser.com/questions/250554/why-does-my-system-hang-when-i-run-ps-w-and-possibly-other-commands


• https://rachelbythebay.com/w/2014/10/27/ps/

swap ps proc

share|improve this question

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Could you post your kernel version and the contents of /proc/$PID/stack of the hanging cat process?
  
  – Danila Kiver
  Jun 30 '18 at 9:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I upvote for the question AND the links, very useful.
  
  – Massimo
  Sep 14 '18 at 17:07
  

  
  
  
  

add a comment | 

4

1

I encountered this strange behavior yesterday on one of our servers. ps, pgrep and htop (on startup) were very slow. strace ps showed that read('/proc/$pid/cmdline) took several seconds on some processes. Why did this happen?

Some observations:

• The processes executable was on NFS


• The processes (about 20+) were doing unlink and symlink operations on files also on NFS, in parallel


• They're forked from the same parent process


• There're 80GB of RAM available (mostly cached), but swap (only 4GB) is in full use


• I run while true; do cat /proc/$pid/status; sleep .1; done, cat returned immediately if State is S or R, but took several seconds when State is D
  

I did some Google'ing and found some SO answers suggesting that when State is D, reading /proc/$pid/cmdline would stall. Is that true? And how does that work? Why was /proc/$pid/cmdline, which was set before the program started, affected by what it was doing after that?

• https://unix.stackexchange.com/a/109864


• https://superuser.com/questions/250554/why-does-my-system-hang-when-i-run-ps-w-and-possibly-other-commands


• https://rachelbythebay.com/w/2014/10/27/ps/

swap ps proc

share|improve this question

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  Could you post your kernel version and the contents of /proc/$PID/stack of the hanging cat process?
  
  – Danila Kiver
  Jun 30 '18 at 9:05
  

  
  
  
  


• 
  
  
  

  
  

  
  
  
  
  
  

  
  
  I upvote for the question AND the links, very useful.
  
  – Massimo
  Sep 14 '18 at 17:07
  

  
  
  
  

add a comment | 

I encountered this strange behavior yesterday on one of our servers. ps, pgrep and htop (on startup) were very slow. strace ps showed that read('/proc/$pid/cmdline) took several seconds on some processes. Why did this happen?

Some observations:

• The processes executable was on NFS


• The processes (about 20+) were doing unlink and symlink operations on files also on NFS, in parallel


• They're forked from the same parent process


• There're 80GB of RAM available (mostly cached), but swap (only 4GB) is in full use


• I run while true; do cat /proc/$pid/status; sleep .1; done, cat returned immediately if State is S or R, but took several seconds when State is D
  

I did some Google'ing and found some SO answers suggesting that when State is D, reading /proc/$pid/cmdline would stall. Is that true? And how does that work? Why was /proc/$pid/cmdline, which was set before the program started, affected by what it was doing after that?

• https://unix.stackexchange.com/a/109864


• https://superuser.com/questions/250554/why-does-my-system-hang-when-i-run-ps-w-and-possibly-other-commands


• https://rachelbythebay.com/w/2014/10/27/ps/

swap ps proc

share|improve this question

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

share|improve this question

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

share|improve this question

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

asked Jun 25 '18 at 3:22

Hai Luong DongHai Luong Dong

17611 silver badge55 bronze badges

1 Answer
1

active

oldest

votes

1

Same here, reading /proc/$pid/cmdline for a special $pid was very slow, even when State is R. And thanks to the links above which point out it might related to NUMA, I found out it caused by numad moving processes from nodes to nodes, this is from /var/log/numad.log :

Thu Jul 18 20:06:41 2019: Advising pid 9565 ($name) move from nodes (0-1) to nodes (1)

Thu Jul 18 20:06:45 2019: PID 9565 moved to node(s) 1 in 3.91 seconds

Thu Jul 18 20:11:50 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:12:00 2019: PID 9565 moved to node(s) 1 in 9.72 seconds

Thu Jul 18 20:17:05 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:17:23 2019: PID 9565 moved to node(s) 1 in 17.85 seconds

Thu Jul 18 20:22:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:22:51 2019: PID 9565 moved to node(s) 1 in 22.73 seconds

Thu Jul 18 20:27:56 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:28:23 2019: PID 9565 moved to node(s) 1 in 26.88 seconds

Thu Jul 18 20:33:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:33:44 2019: PID 9565 moved to node(s) 1 in 15.49 seconds

When moving processes, read cmdline is slow, because cmdline is from user space and kernel need to lock(?) the page and read.

I guess the later move from the same node1 to node1 is needed because the process 9565 was on node1, but it might use remote memory.

% numactl -s

policy: default

preferred node: current

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

cpubind: 0 1

nodebind: 0 1

membind: 0 1

Thanks.

share|improve this answer

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f451673%2fhow-can-cat-proc-pid-cmdline-take-several-seconds%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

Same here, reading /proc/$pid/cmdline for a special $pid was very slow, even when State is R. And thanks to the links above which point out it might related to NUMA, I found out it caused by numad moving processes from nodes to nodes, this is from /var/log/numad.log :

Thu Jul 18 20:06:41 2019: Advising pid 9565 ($name) move from nodes (0-1) to nodes (1)

Thu Jul 18 20:06:45 2019: PID 9565 moved to node(s) 1 in 3.91 seconds

Thu Jul 18 20:11:50 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:12:00 2019: PID 9565 moved to node(s) 1 in 9.72 seconds

Thu Jul 18 20:17:05 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:17:23 2019: PID 9565 moved to node(s) 1 in 17.85 seconds

Thu Jul 18 20:22:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:22:51 2019: PID 9565 moved to node(s) 1 in 22.73 seconds

Thu Jul 18 20:27:56 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:28:23 2019: PID 9565 moved to node(s) 1 in 26.88 seconds

Thu Jul 18 20:33:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:33:44 2019: PID 9565 moved to node(s) 1 in 15.49 seconds

When moving processes, read cmdline is slow, because cmdline is from user space and kernel need to lock(?) the page and read.

I guess the later move from the same node1 to node1 is needed because the process 9565 was on node1, but it might use remote memory.

% numactl -s

policy: default

preferred node: current

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

cpubind: 0 1

nodebind: 0 1

membind: 0 1

Thanks.

share|improve this answer

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

1

Same here, reading /proc/$pid/cmdline for a special $pid was very slow, even when State is R. And thanks to the links above which point out it might related to NUMA, I found out it caused by numad moving processes from nodes to nodes, this is from /var/log/numad.log :

Thu Jul 18 20:06:41 2019: Advising pid 9565 ($name) move from nodes (0-1) to nodes (1)

Thu Jul 18 20:06:45 2019: PID 9565 moved to node(s) 1 in 3.91 seconds

Thu Jul 18 20:11:50 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:12:00 2019: PID 9565 moved to node(s) 1 in 9.72 seconds

Thu Jul 18 20:17:05 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:17:23 2019: PID 9565 moved to node(s) 1 in 17.85 seconds

Thu Jul 18 20:22:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:22:51 2019: PID 9565 moved to node(s) 1 in 22.73 seconds

Thu Jul 18 20:27:56 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:28:23 2019: PID 9565 moved to node(s) 1 in 26.88 seconds

Thu Jul 18 20:33:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:33:44 2019: PID 9565 moved to node(s) 1 in 15.49 seconds

When moving processes, read cmdline is slow, because cmdline is from user space and kernel need to lock(?) the page and read.

I guess the later move from the same node1 to node1 is needed because the process 9565 was on node1, but it might use remote memory.

% numactl -s

policy: default

preferred node: current

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

cpubind: 0 1

nodebind: 0 1

membind: 0 1

Thanks.

share|improve this answer

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

add a comment | 

Same here, reading /proc/$pid/cmdline for a special $pid was very slow, even when State is R. And thanks to the links above which point out it might related to NUMA, I found out it caused by numad moving processes from nodes to nodes, this is from /var/log/numad.log :

Thu Jul 18 20:06:41 2019: Advising pid 9565 ($name) move from nodes (0-1) to nodes (1)

Thu Jul 18 20:06:45 2019: PID 9565 moved to node(s) 1 in 3.91 seconds

Thu Jul 18 20:11:50 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:12:00 2019: PID 9565 moved to node(s) 1 in 9.72 seconds

Thu Jul 18 20:17:05 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:17:23 2019: PID 9565 moved to node(s) 1 in 17.85 seconds

Thu Jul 18 20:22:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:22:51 2019: PID 9565 moved to node(s) 1 in 22.73 seconds

Thu Jul 18 20:27:56 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:28:23 2019: PID 9565 moved to node(s) 1 in 26.88 seconds

Thu Jul 18 20:33:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:33:44 2019: PID 9565 moved to node(s) 1 in 15.49 seconds

When moving processes, read cmdline is slow, because cmdline is from user space and kernel need to lock(?) the page and read.

I guess the later move from the same node1 to node1 is needed because the process 9565 was on node1, but it might use remote memory.

% numactl -s

policy: default

preferred node: current

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

cpubind: 0 1

nodebind: 0 1

membind: 0 1

Thanks.

share|improve this answer

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

Thu Jul 18 20:06:41 2019: Advising pid 9565 ($name) move from nodes (0-1) to nodes (1)

Thu Jul 18 20:06:45 2019: PID 9565 moved to node(s) 1 in 3.91 seconds

Thu Jul 18 20:11:50 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:12:00 2019: PID 9565 moved to node(s) 1 in 9.72 seconds

Thu Jul 18 20:17:05 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:17:23 2019: PID 9565 moved to node(s) 1 in 17.85 seconds

Thu Jul 18 20:22:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:22:51 2019: PID 9565 moved to node(s) 1 in 22.73 seconds

Thu Jul 18 20:27:56 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:28:23 2019: PID 9565 moved to node(s) 1 in 26.88 seconds

Thu Jul 18 20:33:28 2019: Advising pid 9565 ($name) move from nodes (1) to nodes (1)

Thu Jul 18 20:33:44 2019: PID 9565 moved to node(s) 1 in 15.49 seconds

% numactl -s

policy: default

preferred node: current

physcpubind: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

cpubind: 0 1

nodebind: 0 1

membind: 0 1

share|improve this answer

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge

New contributor

Opera Wang is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

answered 1 hour ago

Opera WangOpera Wang

1111 bronze badge