Systemd kills service immediately after startSystemd and process spawningWhy is systemd stopping service...
Is it OK to look at the list of played moves during the game to determine the status of the 50 move rule?
Is the default 512 byte physical sector size appropriate for SSD disks under Linux?
One word for 'the thing that attracts me'?
Why did Nick Fury not hesitate in blowing up the plane he thought was carrying a nuke?
A nasty indefinite integral
Would this be a dangerous impeller to use for a drone?
How to test if argument is a single space?
Ribbon Cable Cross Talk - Is there a fix after the fact?
Is there an idiom that means that you are in a very strong negotiation position in a negotiation?
How to make Flex Markers appear in Logic Pro X?
What pc resources are used when bruteforcing?
To exponential digit growth and beyond!
Is there a word for pant sleeves?
How many wires should be in a new thermostat cable?
How did the Allies achieve air superiority on Sicily?
Unary Enumeration
Which values for voltage divider
What defines a person who is circumcised "of the heart"?
Keeping the dodos out of the field
"Official wife" or "Formal wife"?
Is it normal to "extract a paper" from a master thesis?
size of pointers and architecture
Is it safe to redirect stdout and stderr to the same file without file descriptor copies?
Efficient Algorithms for Destroyed Document Reconstruction
Systemd kills service immediately after start
Systemd and process spawningWhy is systemd stopping service immediately after it is started?systemd failing to recognize pid fileCentOS 7 - why the service file is not working to run my bash script?Systemd timeout because it doesn't detect daemon forkingsystemctl fails to accurately report status of daemons if they were started via mechanism other than systemctlIssues creating and enabling custom systemd services for ddclientSerivce Restarts every 90 secondsWhy forking is used in a unit file of a service?Systemd service starts only one of two processes, but only on OS rebootAdding a systemd .service (Debian)Force systemd to check status of service after n secondsmysql service restarted during user being connected lead to failing serviceHow to start a systemd service after mount commandSystemd service starts only one of two processes, but only on OS rebootsystemd PrivateTmp/JoinsNamespaceOfWhy is systemd stopping service immediately after it is started?Instruct to execute an unit after completing another unit successfullyStopping systemd unit together with another. Starting works
.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}
I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.
When I use that ExecStart directive all working fine.
ExecStart=/var/ossec/bin/ossec-control start
But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'
If i make another small change service will receive SIG 15 after 20 seconds.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'
So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.
How can I solve this problem?
centos rhel systemd
add a comment |
I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.
When I use that ExecStart directive all working fine.
ExecStart=/var/ossec/bin/ossec-control start
But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'
If i make another small change service will receive SIG 15 after 20 seconds.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'
So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.
How can I solve this problem?
centos rhel systemd
1
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04
add a comment |
I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.
When I use that ExecStart directive all working fine.
ExecStart=/var/ossec/bin/ossec-control start
But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'
If i make another small change service will receive SIG 15 after 20 seconds.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'
So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.
How can I solve this problem?
centos rhel systemd
I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.
When I use that ExecStart directive all working fine.
ExecStart=/var/ossec/bin/ossec-control start
But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'
If i make another small change service will receive SIG 15 after 20 seconds.
ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'
So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.
How can I solve this problem?
centos rhel systemd
centos rhel systemd
asked May 4 '15 at 10:40
Daniil SvetlovDaniil Svetlov
46112
46112
1
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04
add a comment |
1
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04
1
1
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04
add a comment |
3 Answers
3
active
oldest
votes
readiness protocol mismatch
As Wieland implied, the Type
of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple
service is assumed to be immediately ready. A forking
service is taken to be ready after its initial process forks a child and then exits. A dbus
service is taken to be ready when a server appears on the Desktop Bus. And so forth.
If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.
You're doing exactly this.
First of all, the simple stuff: sh -c
doesn't match Type=simple
or Type=forking
.
In the simple
protocol, the initial process is taken to be the service process. But in fact a sh -c
wrapper runs the actual service program as a child process. So MAINPID
goes wrong and ExecReload
stops working, for starters. When using Type=simple
, one must either use sh -c 'exec …'
or not use sh -c
in the first place. The latter is more often the correct course than some people think.
sh -c
doesn't match Type=forking
either. The readiness protocol for a forking
service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.
the unnecessary horror that is ossec-control
Which brings us to the complex stuff: that ossec-control
script.
It turns out that it's a System 5 rc
script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc
scripts that attempts to manage a whole set of server processes in one single script, with for
loops, race conditions, arbitrary sleep
s to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable
and disable
verbs.
So when you /bin/sh -c '/var/ossec/bin/ossec-control start'
what happens is that:
- systemd forks what it expects to be the service process.
- That's the shell, which forks
ossec-control
. - That in turn forks between 4 and 10 grandchildren.
- The grandchildren all fork and exit in turn.
- The great-grandchildren all fork and exit in parallel.
ossec-control
exits.- The first shell exits.
- The service processes were the great-great-grandchildren, but because this way of working matches neither the
forking
nor thesimple
readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.
None of this horror is actually necessary under systemd at all. None of it.
a systemd template service unit
Instead, one writes a very simple template unit:
[Unit]
Description=The OSSEC HIDS %i server
After=network.target
[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f
[Install]
WantedBy=multi-user.target
Save this this as /etc/systemd/system/ossec@.service
.
The various actual services are instantiations of this template, named:
ossec@dbd.service
ossec@agentlessd.service
ossec@csyslogd.service
ossec@execd.service
ossec@agentd.service
ossec@logcollector.service
ossec@syscheckd.service
ossec@maild.service
ossec@analysisd.service
ossec@remoted.service
ossec@monitord.service
Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.
systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord
Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u
. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.
By the way: Type=simple
and the -f
option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit
, and these here are not such cases either. But that's what the forking
type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.
Further reading
- Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, whichRequires=
all the needed instances, and then setPartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.
– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
add a comment |
Keep Type=forking and give a pid file location if start service/app is maintaining any pid.
[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service
[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog
[Install]
WantedBy=multi-user.target
Alias=webapps
New contributor
add a comment |
Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.
JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "106"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f200280%2fsystemd-kills-service-immediately-after-start%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
3 Answers
3
active
oldest
votes
3 Answers
3
active
oldest
votes
active
oldest
votes
active
oldest
votes
readiness protocol mismatch
As Wieland implied, the Type
of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple
service is assumed to be immediately ready. A forking
service is taken to be ready after its initial process forks a child and then exits. A dbus
service is taken to be ready when a server appears on the Desktop Bus. And so forth.
If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.
You're doing exactly this.
First of all, the simple stuff: sh -c
doesn't match Type=simple
or Type=forking
.
In the simple
protocol, the initial process is taken to be the service process. But in fact a sh -c
wrapper runs the actual service program as a child process. So MAINPID
goes wrong and ExecReload
stops working, for starters. When using Type=simple
, one must either use sh -c 'exec …'
or not use sh -c
in the first place. The latter is more often the correct course than some people think.
sh -c
doesn't match Type=forking
either. The readiness protocol for a forking
service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.
the unnecessary horror that is ossec-control
Which brings us to the complex stuff: that ossec-control
script.
It turns out that it's a System 5 rc
script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc
scripts that attempts to manage a whole set of server processes in one single script, with for
loops, race conditions, arbitrary sleep
s to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable
and disable
verbs.
So when you /bin/sh -c '/var/ossec/bin/ossec-control start'
what happens is that:
- systemd forks what it expects to be the service process.
- That's the shell, which forks
ossec-control
. - That in turn forks between 4 and 10 grandchildren.
- The grandchildren all fork and exit in turn.
- The great-grandchildren all fork and exit in parallel.
ossec-control
exits.- The first shell exits.
- The service processes were the great-great-grandchildren, but because this way of working matches neither the
forking
nor thesimple
readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.
None of this horror is actually necessary under systemd at all. None of it.
a systemd template service unit
Instead, one writes a very simple template unit:
[Unit]
Description=The OSSEC HIDS %i server
After=network.target
[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f
[Install]
WantedBy=multi-user.target
Save this this as /etc/systemd/system/ossec@.service
.
The various actual services are instantiations of this template, named:
ossec@dbd.service
ossec@agentlessd.service
ossec@csyslogd.service
ossec@execd.service
ossec@agentd.service
ossec@logcollector.service
ossec@syscheckd.service
ossec@maild.service
ossec@analysisd.service
ossec@remoted.service
ossec@monitord.service
Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.
systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord
Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u
. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.
By the way: Type=simple
and the -f
option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit
, and these here are not such cases either. But that's what the forking
type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.
Further reading
- Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, whichRequires=
all the needed instances, and then setPartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.
– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
add a comment |
readiness protocol mismatch
As Wieland implied, the Type
of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple
service is assumed to be immediately ready. A forking
service is taken to be ready after its initial process forks a child and then exits. A dbus
service is taken to be ready when a server appears on the Desktop Bus. And so forth.
If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.
You're doing exactly this.
First of all, the simple stuff: sh -c
doesn't match Type=simple
or Type=forking
.
In the simple
protocol, the initial process is taken to be the service process. But in fact a sh -c
wrapper runs the actual service program as a child process. So MAINPID
goes wrong and ExecReload
stops working, for starters. When using Type=simple
, one must either use sh -c 'exec …'
or not use sh -c
in the first place. The latter is more often the correct course than some people think.
sh -c
doesn't match Type=forking
either. The readiness protocol for a forking
service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.
the unnecessary horror that is ossec-control
Which brings us to the complex stuff: that ossec-control
script.
It turns out that it's a System 5 rc
script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc
scripts that attempts to manage a whole set of server processes in one single script, with for
loops, race conditions, arbitrary sleep
s to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable
and disable
verbs.
So when you /bin/sh -c '/var/ossec/bin/ossec-control start'
what happens is that:
- systemd forks what it expects to be the service process.
- That's the shell, which forks
ossec-control
. - That in turn forks between 4 and 10 grandchildren.
- The grandchildren all fork and exit in turn.
- The great-grandchildren all fork and exit in parallel.
ossec-control
exits.- The first shell exits.
- The service processes were the great-great-grandchildren, but because this way of working matches neither the
forking
nor thesimple
readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.
None of this horror is actually necessary under systemd at all. None of it.
a systemd template service unit
Instead, one writes a very simple template unit:
[Unit]
Description=The OSSEC HIDS %i server
After=network.target
[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f
[Install]
WantedBy=multi-user.target
Save this this as /etc/systemd/system/ossec@.service
.
The various actual services are instantiations of this template, named:
ossec@dbd.service
ossec@agentlessd.service
ossec@csyslogd.service
ossec@execd.service
ossec@agentd.service
ossec@logcollector.service
ossec@syscheckd.service
ossec@maild.service
ossec@analysisd.service
ossec@remoted.service
ossec@monitord.service
Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.
systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord
Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u
. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.
By the way: Type=simple
and the -f
option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit
, and these here are not such cases either. But that's what the forking
type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.
Further reading
- Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, whichRequires=
all the needed instances, and then setPartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.
– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
add a comment |
readiness protocol mismatch
As Wieland implied, the Type
of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple
service is assumed to be immediately ready. A forking
service is taken to be ready after its initial process forks a child and then exits. A dbus
service is taken to be ready when a server appears on the Desktop Bus. And so forth.
If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.
You're doing exactly this.
First of all, the simple stuff: sh -c
doesn't match Type=simple
or Type=forking
.
In the simple
protocol, the initial process is taken to be the service process. But in fact a sh -c
wrapper runs the actual service program as a child process. So MAINPID
goes wrong and ExecReload
stops working, for starters. When using Type=simple
, one must either use sh -c 'exec …'
or not use sh -c
in the first place. The latter is more often the correct course than some people think.
sh -c
doesn't match Type=forking
either. The readiness protocol for a forking
service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.
the unnecessary horror that is ossec-control
Which brings us to the complex stuff: that ossec-control
script.
It turns out that it's a System 5 rc
script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc
scripts that attempts to manage a whole set of server processes in one single script, with for
loops, race conditions, arbitrary sleep
s to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable
and disable
verbs.
So when you /bin/sh -c '/var/ossec/bin/ossec-control start'
what happens is that:
- systemd forks what it expects to be the service process.
- That's the shell, which forks
ossec-control
. - That in turn forks between 4 and 10 grandchildren.
- The grandchildren all fork and exit in turn.
- The great-grandchildren all fork and exit in parallel.
ossec-control
exits.- The first shell exits.
- The service processes were the great-great-grandchildren, but because this way of working matches neither the
forking
nor thesimple
readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.
None of this horror is actually necessary under systemd at all. None of it.
a systemd template service unit
Instead, one writes a very simple template unit:
[Unit]
Description=The OSSEC HIDS %i server
After=network.target
[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f
[Install]
WantedBy=multi-user.target
Save this this as /etc/systemd/system/ossec@.service
.
The various actual services are instantiations of this template, named:
ossec@dbd.service
ossec@agentlessd.service
ossec@csyslogd.service
ossec@execd.service
ossec@agentd.service
ossec@logcollector.service
ossec@syscheckd.service
ossec@maild.service
ossec@analysisd.service
ossec@remoted.service
ossec@monitord.service
Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.
systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord
Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u
. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.
By the way: Type=simple
and the -f
option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit
, and these here are not such cases either. But that's what the forking
type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.
Further reading
- Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
readiness protocol mismatch
As Wieland implied, the Type
of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple
service is assumed to be immediately ready. A forking
service is taken to be ready after its initial process forks a child and then exits. A dbus
service is taken to be ready when a server appears on the Desktop Bus. And so forth.
If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.
You're doing exactly this.
First of all, the simple stuff: sh -c
doesn't match Type=simple
or Type=forking
.
In the simple
protocol, the initial process is taken to be the service process. But in fact a sh -c
wrapper runs the actual service program as a child process. So MAINPID
goes wrong and ExecReload
stops working, for starters. When using Type=simple
, one must either use sh -c 'exec …'
or not use sh -c
in the first place. The latter is more often the correct course than some people think.
sh -c
doesn't match Type=forking
either. The readiness protocol for a forking
service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.
the unnecessary horror that is ossec-control
Which brings us to the complex stuff: that ossec-control
script.
It turns out that it's a System 5 rc
script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc
scripts that attempts to manage a whole set of server processes in one single script, with for
loops, race conditions, arbitrary sleep
s to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable
and disable
verbs.
So when you /bin/sh -c '/var/ossec/bin/ossec-control start'
what happens is that:
- systemd forks what it expects to be the service process.
- That's the shell, which forks
ossec-control
. - That in turn forks between 4 and 10 grandchildren.
- The grandchildren all fork and exit in turn.
- The great-grandchildren all fork and exit in parallel.
ossec-control
exits.- The first shell exits.
- The service processes were the great-great-grandchildren, but because this way of working matches neither the
forking
nor thesimple
readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.
None of this horror is actually necessary under systemd at all. None of it.
a systemd template service unit
Instead, one writes a very simple template unit:
[Unit]
Description=The OSSEC HIDS %i server
After=network.target
[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f
[Install]
WantedBy=multi-user.target
Save this this as /etc/systemd/system/ossec@.service
.
The various actual services are instantiations of this template, named:
ossec@dbd.service
ossec@agentlessd.service
ossec@csyslogd.service
ossec@execd.service
ossec@agentd.service
ossec@logcollector.service
ossec@syscheckd.service
ossec@maild.service
ossec@analysisd.service
ossec@remoted.service
ossec@monitord.service
Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.
systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord
Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u
. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.
By the way: Type=simple
and the -f
option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit
, and these here are not such cases either. But that's what the forking
type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.
Further reading
- Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.
edited Sep 26 '17 at 2:55
answered May 4 '15 at 17:55
JdeBPJdeBP
39.5k482191
39.5k482191
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, whichRequires=
all the needed instances, and then setPartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.
– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
add a comment |
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, whichRequires=
all the needed instances, and then setPartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.
– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
1
1
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which
Requires=
all the needed instances, and then set PartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.– intelfx
May 4 '15 at 18:45
Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which
Requires=
all the needed instances, and then set PartOf=ossec.target
in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.– intelfx
May 4 '15 at 18:45
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
@JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.
– Daniil Svetlov
May 4 '15 at 22:14
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
What's the reason for using /usr/bin/env as a wrapper?
– Marius Gedminas
Mar 30 at 16:18
add a comment |
Keep Type=forking and give a pid file location if start service/app is maintaining any pid.
[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service
[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog
[Install]
WantedBy=multi-user.target
Alias=webapps
New contributor
add a comment |
Keep Type=forking and give a pid file location if start service/app is maintaining any pid.
[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service
[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog
[Install]
WantedBy=multi-user.target
Alias=webapps
New contributor
add a comment |
Keep Type=forking and give a pid file location if start service/app is maintaining any pid.
[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service
[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog
[Install]
WantedBy=multi-user.target
Alias=webapps
New contributor
Keep Type=forking and give a pid file location if start service/app is maintaining any pid.
[Unit]
Description="Run app on boot"
After=network.target syslog.target auditd.service
[Service]
Type=forking
PIDFile=/var/run/apache2/apache2.pid
ExecStart=/etc/init.d/apache2 start
ExecStop=/etc/init.d/apache2 stop
StandardOutput=syslog
StandardError=syslog
Restart=on-failure
SyslogIdentifier=webappslog
[Install]
WantedBy=multi-user.target
Alias=webapps
New contributor
New contributor
answered 37 mins ago
RaushanRaushan
1
1
New contributor
New contributor
add a comment |
add a comment |
Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.
JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
add a comment |
Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.
JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
add a comment |
Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.
JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.
Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.
JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.
answered Sep 20 '16 at 7:07
JohnJohn
11
11
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
add a comment |
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.
– JoostM
Nov 11 '18 at 10:17
add a comment |
Thanks for contributing an answer to Unix & Linux Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f200280%2fsystemd-kills-service-immediately-after-start%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
What's the Type of the service?
– Wieland
May 4 '15 at 13:01
@Wieland, I was try simple and forking, but result is still the same.
– Daniil Svetlov
May 4 '15 at 16:04