How to increase nvme_core.io_timeout on my c5 EC2 instacnceixgbe: increase rx performanceAWS EC2...
Is it possible to have 2 different but equal size real number sets that have the same mean and standard deviation?
How creative should the DM let an artificer be in terms of what they can build?
How can I remove material from this wood beam?
What is the color of artificial intelligence?
Longest bridge/tunnel that can be cycled over/through?
Printing Pascal’s triangle for n number of rows in Python
Increase speed altering column on large table to NON NULL
Why not invest in precious metals?
Neat redistribution of association values
The Frozen Wastes
What does a topology do, and what makes a particular topology the 'right' one?
What differences exist between adamantine and adamantite in all editions of D&D?
With Ubuntu 18.04, how can I have a hot corner that locks the computer?
How to learn Linux system internals
Which languages would be most useful in Europe at the end of the 19th century?
Complicated equation, make x the subject
What are neighboring ports?
Why can I traceroute to this IP address, but not ping?
Which is the better way to call a method that is only available to one class that implements an interface but not the other one?
Why does this query, missing a FROM clause, not error out?
I have a problematic assistant manager, but I can't fire him
Why did Intel abandon unified CPU cache?
What is the meaning of the Russian idiom "to taste tuna" ("отведать тунца")?
How to communicate to my GM that not being allowed to use stealth isn't fun for me?
How to increase nvme_core.io_timeout on my c5 EC2 instacnce
ixgbe: increase rx performanceAWS EC2 infrastructureAWS EC2 ethernet irq sharing issueSSH ec2 goes to private IPinstall chrome on aws ec2 linux server?Run a script at each startup of my EC2 instanceCan't connect to AWS EC2 via SSHSSH Error on AWS EC2Launch ec2 instance using boto3 with security groupMount EFS drive on EC2 instance
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
We have mesos cluster where we're running centos7 c5 instances on aws. The kernel version is the latest 4.16.1-1.
In c5 instance type the volumes uses nvme drivers. The nvme volumes seems to have a behavior as mentioned here where if there is an io timeout on a volume, the volume mount becomes read only and no further writes can happen. So if there is heavy read-write operations on your device like on root drive then after the io timeout no further writes can happen so its dangerous.
In AWS documentation it mentioned to set an io timeout as high as possible and it seems to be 4294967295 sec.
AWS doc specify that default io timeout is 30sec, but it is max 255 sec for kernel prior to 4.15 version and 4294967295 sec for kernel 4.15+. As we have latest 4.16.1 kernel we should set it to max 4294967295 sec.
But when I try to set the nvme_core.io_timeout parameter to the max value, it didn't get refelected. I tried this
sh-4.2# modprobe nvme_core io_timeout=123457
sh-4.2# cat /sys/module/nvme_core/parameters/io_timeout
30
sh-4.2#
What is the correct way to set nvme_core.io_timeout I tried lot of other things like
- setting it in
/etc/default/grubfile - sysctl command
- Overriding
/sys/module/nvme_core/parameters/io_timeoutfile
But Nothing helped.
linux kernel aws modprobe nvme
asked May 23 '18 at 9:21
mchawremchawre
124210
bumped to the homepage by Community♦ 45 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
We have mesos cluster where we're running centos7 c5 instances on aws. The kernel version is the latest 4.16.1-1.
In c5 instance type the volumes uses nvme drivers. The nvme volumes seems to have a behavior as mentioned here where if there is an io timeout on a volume, the volume mount becomes read only and no further writes can happen. So if there is heavy read-write operations on your device like on root drive then after the io timeout no further writes can happen so its dangerous.
In AWS documentation it mentioned to set an io timeout as high as possible and it seems to be 4294967295 sec.
AWS doc specify that default io timeout is 30sec, but it is max 255 sec for kernel prior to 4.15 version and 4294967295 sec for kernel 4.15+. As we have latest 4.16.1 kernel we should set it to max 4294967295 sec.
But when I try to set the nvme_core.io_timeout parameter to the max value, it didn't get refelected. I tried this
sh-4.2# modprobe nvme_core io_timeout=123457
sh-4.2# cat /sys/module/nvme_core/parameters/io_timeout
30
sh-4.2#
What is the correct way to set nvme_core.io_timeout I tried lot of other things like
- setting it in
/etc/default/grubfile - sysctl command
- Overriding
/sys/module/nvme_core/parameters/io_timeoutfile
But Nothing helped.
linux kernel aws modprobe nvme
asked May 23 '18 at 9:21
mchawremchawre
124210
bumped to the homepage by Community♦ 45 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
We have mesos cluster where we're running centos7 c5 instances on aws. The kernel version is the latest 4.16.1-1.
In c5 instance type the volumes uses nvme drivers. The nvme volumes seems to have a behavior as mentioned here where if there is an io timeout on a volume, the volume mount becomes read only and no further writes can happen. So if there is heavy read-write operations on your device like on root drive then after the io timeout no further writes can happen so its dangerous.
In AWS documentation it mentioned to set an io timeout as high as possible and it seems to be 4294967295 sec.
AWS doc specify that default io timeout is 30sec, but it is max 255 sec for kernel prior to 4.15 version and 4294967295 sec for kernel 4.15+. As we have latest 4.16.1 kernel we should set it to max 4294967295 sec.
But when I try to set the nvme_core.io_timeout parameter to the max value, it didn't get refelected. I tried this
sh-4.2# modprobe nvme_core io_timeout=123457
sh-4.2# cat /sys/module/nvme_core/parameters/io_timeout
30
sh-4.2#
What is the correct way to set nvme_core.io_timeout I tried lot of other things like
- setting it in
/etc/default/grubfile - sysctl command
- Overriding
/sys/module/nvme_core/parameters/io_timeoutfile
But Nothing helped.
linux kernel aws modprobe nvme
asked May 23 '18 at 9:21
mchawremchawre
124210
We have mesos cluster where we're running centos7 c5 instances on aws. The kernel version is the latest 4.16.1-1.
In c5 instance type the volumes uses nvme drivers. The nvme volumes seems to have a behavior as mentioned here where if there is an io timeout on a volume, the volume mount becomes read only and no further writes can happen. So if there is heavy read-write operations on your device like on root drive then after the io timeout no further writes can happen so its dangerous.
In AWS documentation it mentioned to set an io timeout as high as possible and it seems to be 4294967295 sec.
AWS doc specify that default io timeout is 30sec, but it is max 255 sec for kernel prior to 4.15 version and 4294967295 sec for kernel 4.15+. As we have latest 4.16.1 kernel we should set it to max 4294967295 sec.
But when I try to set the nvme_core.io_timeout parameter to the max value, it didn't get refelected. I tried this
sh-4.2# modprobe nvme_core io_timeout=123457
sh-4.2# cat /sys/module/nvme_core/parameters/io_timeout
30
sh-4.2#
What is the correct way to set nvme_core.io_timeout I tried lot of other things like
- setting it in
/etc/default/grubfile - sysctl command
- Overriding
/sys/module/nvme_core/parameters/io_timeoutfile
But Nothing helped.
linux kernel aws modprobe nvme
linux kernel aws modprobe nvme
asked May 23 '18 at 9:21
mchawremchawre
124210
asked May 23 '18 at 9:21
mchawremchawre
124210
edited May 24 '18 at 9:07
mchawre
asked May 23 '18 at 9:21
mchawremchawre
124210
asked May 23 '18 at 9:21
mchawremchawre
124210
asked May 23 '18 at 9:21
mchawremchawre
124210
124210
bumped to the homepage by Community♦ 45 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
bumped to the homepage by Community♦ 45 mins ago
This question has answers that may be good or bad; the system has marked it active so that they can be reviewed.
add a comment |
add a comment |
1 Answer
1
active
oldest
votes
Based on my own experimentation, we do this while building our AMIs.
cp /etc/default/grub /tmp/grub
cat >>/tmp/grub <<'EOF'
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} nvme_core.io_timeout=255"
EOF
sudo mv /tmp/grub /etc/default/grub
sudo update-grub
Then create an AMI from the instance. When you start a new EC2 instance from the AMI, it comes up with the correct setting.
Obviously this can be modify to set any kernel parameter.
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f445519%2fhow-to-increase-nvme-core-io-timeout-on-my-c5-ec2-instacnce%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Based on my own experimentation, we do this while building our AMIs.
cp /etc/default/grub /tmp/grub
cat >>/tmp/grub <<'EOF'
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} nvme_core.io_timeout=255"
EOF
sudo mv /tmp/grub /etc/default/grub
sudo update-grub
Then create an AMI from the instance. When you start a new EC2 instance from the AMI, it comes up with the correct setting.
Obviously this can be modify to set any kernel parameter.
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
add a comment |
Based on my own experimentation, we do this while building our AMIs.
cp /etc/default/grub /tmp/grub
cat >>/tmp/grub <<'EOF'
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} nvme_core.io_timeout=255"
EOF
sudo mv /tmp/grub /etc/default/grub
sudo update-grub
Then create an AMI from the instance. When you start a new EC2 instance from the AMI, it comes up with the correct setting.
Obviously this can be modify to set any kernel parameter.
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
add a comment |
Based on my own experimentation, we do this while building our AMIs.
cp /etc/default/grub /tmp/grub
cat >>/tmp/grub <<'EOF'
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} nvme_core.io_timeout=255"
EOF
sudo mv /tmp/grub /etc/default/grub
sudo update-grub
Then create an AMI from the instance. When you start a new EC2 instance from the AMI, it comes up with the correct setting.
Obviously this can be modify to set any kernel parameter.
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
Based on my own experimentation, we do this while building our AMIs.
cp /etc/default/grub /tmp/grub
cat >>/tmp/grub <<'EOF'
GRUB_CMDLINE_LINUX="${GRUB_CMDLINE_LINUX} nvme_core.io_timeout=255"
EOF
sudo mv /tmp/grub /etc/default/grub
sudo update-grub
Then create an AMI from the instance. When you start a new EC2 instance from the AMI, it comes up with the correct setting.
Obviously this can be modify to set any kernel parameter.
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
answered Aug 17 '18 at 13:15
Ben Butler-ColeBen Butler-Cole
1233
1233
add a comment |
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f445519%2fhow-to-increase-nvme-core-io-timeout-on-my-c5-ec2-instacnce%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
