Systemd kills service immediately after startSystemd and process spawningWhy is systemd stopping service...

Is it OK to look at the list of played moves during the game to determine the status of the 50 move rule?

Is the default 512 byte physical sector size appropriate for SSD disks under Linux?

One word for 'the thing that attracts me'?

Why did Nick Fury not hesitate in blowing up the plane he thought was carrying a nuke?

A nasty indefinite integral

Would this be a dangerous impeller to use for a drone?

How to test if argument is a single space?

Ribbon Cable Cross Talk - Is there a fix after the fact?

Is there an idiom that means that you are in a very strong negotiation position in a negotiation?

How to make Flex Markers appear in Logic Pro X?

What pc resources are used when bruteforcing?

To exponential digit growth and beyond!

Is there a word for pant sleeves?

How many wires should be in a new thermostat cable?

How did the Allies achieve air superiority on Sicily?

Unary Enumeration

Which values for voltage divider

What defines a person who is circumcised "of the heart"?

Keeping the dodos out of the field

"Official wife" or "Formal wife"?

Is it normal to "extract a paper" from a master thesis?

size of pointers and architecture

Is it safe to redirect stdout and stderr to the same file without file descriptor copies?

Efficient Algorithms for Destroyed Document Reconstruction



Systemd kills service immediately after start


Systemd and process spawningWhy is systemd stopping service immediately after it is started?systemd failing to recognize pid fileCentOS 7 - why the service file is not working to run my bash script?Systemd timeout because it doesn't detect daemon forkingsystemctl fails to accurately report status of daemons if they were started via mechanism other than systemctlIssues creating and enabling custom systemd services for ddclientSerivce Restarts every 90 secondsWhy forking is used in a unit file of a service?Systemd service starts only one of two processes, but only on OS rebootAdding a systemd .service (Debian)Force systemd to check status of service after n secondsmysql service restarted during user being connected lead to failing serviceHow to start a systemd service after mount commandSystemd service starts only one of two processes, but only on OS rebootsystemd PrivateTmp/JoinsNamespaceOfWhy is systemd stopping service immediately after it is started?Instruct to execute an unit after completing another unit successfullyStopping systemd unit together with another. Starting works






.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}







9















I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.



When I use that ExecStart directive all working fine.



ExecStart=/var/ossec/bin/ossec-control start


But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'


If i make another small change service will receive SIG 15 after 20 seconds.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'


So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.



How can I solve this problem?










share|improve this question


















  • 1





    What's the Type of the service?

    – Wieland
    May 4 '15 at 13:01











  • @Wieland, I was try simple and forking, but result is still the same.

    – Daniil Svetlov
    May 4 '15 at 16:04


















9















I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.



When I use that ExecStart directive all working fine.



ExecStart=/var/ossec/bin/ossec-control start


But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'


If i make another small change service will receive SIG 15 after 20 seconds.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'


So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.



How can I solve this problem?










share|improve this question


















  • 1





    What's the Type of the service?

    – Wieland
    May 4 '15 at 13:01











  • @Wieland, I was try simple and forking, but result is still the same.

    – Daniil Svetlov
    May 4 '15 at 16:04














9












9








9


7






I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.



When I use that ExecStart directive all working fine.



ExecStart=/var/ossec/bin/ossec-control start


But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'


If i make another small change service will receive SIG 15 after 20 seconds.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'


So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.



How can I solve this problem?










share|improve this question














I'm writing systemd unit file for OSSEC HIDS.
Problem is that when systemd starts service it immediately stops them.



When I use that ExecStart directive all working fine.



ExecStart=/var/ossec/bin/ossec-control start


But when I make small improvement I fine in OSSEC logs, that it receive SIG 15 after start.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start'


If i make another small change service will receive SIG 15 after 20 seconds.



ExecStart=/bin/sh -c '${DIRECTORY}/bin/ossec-control start && sleep 20'


So, I guess, that systemd kills /bin/sh process after service start, and bin/sh then kills OSSEC.



How can I solve this problem?







centos rhel systemd






share|improve this question













share|improve this question











share|improve this question




share|improve this question










asked May 4 '15 at 10:40









Daniil SvetlovDaniil Svetlov

46112




46112








  • 1





    What's the Type of the service?

    – Wieland
    May 4 '15 at 13:01











  • @Wieland, I was try simple and forking, but result is still the same.

    – Daniil Svetlov
    May 4 '15 at 16:04














  • 1





    What's the Type of the service?

    – Wieland
    May 4 '15 at 13:01











  • @Wieland, I was try simple and forking, but result is still the same.

    – Daniil Svetlov
    May 4 '15 at 16:04








1




1





What's the Type of the service?

– Wieland
May 4 '15 at 13:01





What's the Type of the service?

– Wieland
May 4 '15 at 13:01













@Wieland, I was try simple and forking, but result is still the same.

– Daniil Svetlov
May 4 '15 at 16:04





@Wieland, I was try simple and forking, but result is still the same.

– Daniil Svetlov
May 4 '15 at 16:04










3 Answers
3






active

oldest

votes


















32














readiness protocol mismatch



As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.



If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.



You're doing exactly this.



First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.



In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.



sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.



the unnecessary horror that is ossec-control



Which brings us to the complex stuff: that ossec-control script.



It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.



So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:




  1. systemd forks what it expects to be the service process.

  2. That's the shell, which forks ossec-control.

  3. That in turn forks between 4 and 10 grandchildren.

  4. The grandchildren all fork and exit in turn.

  5. The great-grandchildren all fork and exit in parallel.


  6. ossec-control exits.

  7. The first shell exits.

  8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.


None of this horror is actually necessary under systemd at all. None of it.



a systemd template service unit



Instead, one writes a very simple template unit:




[Unit]
Description=The OSSEC HIDS %i server
After=network.target

[Service]
Type=simple
ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

[Install]
WantedBy=multi-user.target


Save this this as /etc/systemd/system/ossec@.service.



The various actual services are instantiations of this template, named:




  • ossec@dbd.service

  • ossec@agentlessd.service

  • ossec@csyslogd.service

  • ossec@execd.service

  • ossec@agentd.service

  • ossec@logcollector.service

  • ossec@syscheckd.service

  • ossec@maild.service

  • ossec@analysisd.service

  • ossec@remoted.service

  • ossec@monitord.service


Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.


 systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord


Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.



By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.



Further reading




  • Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.






share|improve this answer





















  • 1





    Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

    – intelfx
    May 4 '15 at 18:45











  • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

    – Daniil Svetlov
    May 4 '15 at 22:14











  • What's the reason for using /usr/bin/env as a wrapper?

    – Marius Gedminas
    Mar 30 at 16:18



















0














Keep Type=forking and give a pid file location if start service/app is maintaining any pid.



[Unit]

Description="Run app on boot"

After=network.target syslog.target auditd.service


[Service]

Type=forking

PIDFile=/var/run/apache2/apache2.pid

ExecStart=/etc/init.d/apache2 start

ExecStop=/etc/init.d/apache2 stop

StandardOutput=syslog

StandardError=syslog

Restart=on-failure

SyslogIdentifier=webappslog


[Install]

WantedBy=multi-user.target

Alias=webapps






share|improve this answer








New contributor



Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.



























    -4














    Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.



    JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.






    share|improve this answer
























    • What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

      – JoostM
      Nov 11 '18 at 10:17














    Your Answer








    StackExchange.ready(function() {
    var channelOptions = {
    tags: "".split(" "),
    id: "106"
    };
    initTagRenderer("".split(" "), "".split(" "), channelOptions);

    StackExchange.using("externalEditor", function() {
    // Have to fire editor after snippets, if snippets enabled
    if (StackExchange.settings.snippets.snippetsEnabled) {
    StackExchange.using("snippets", function() {
    createEditor();
    });
    }
    else {
    createEditor();
    }
    });

    function createEditor() {
    StackExchange.prepareEditor({
    heartbeatType: 'answer',
    autoActivateHeartbeat: false,
    convertImagesToLinks: false,
    noModals: true,
    showLowRepImageUploadWarning: true,
    reputationToPostImages: null,
    bindNavPrevention: true,
    postfix: "",
    imageUploader: {
    brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
    contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
    allowUrls: true
    },
    onDemand: true,
    discardSelector: ".discard-answer"
    ,immediatelyShowMarkdownHelp:true
    });


    }
    });














    draft saved

    draft discarded


















    StackExchange.ready(
    function () {
    StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f200280%2fsystemd-kills-service-immediately-after-start%23new-answer', 'question_page');
    }
    );

    Post as a guest















    Required, but never shown

























    3 Answers
    3






    active

    oldest

    votes








    3 Answers
    3






    active

    oldest

    votes









    active

    oldest

    votes






    active

    oldest

    votes









    32














    readiness protocol mismatch



    As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.



    If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.



    You're doing exactly this.



    First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.



    In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.



    sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.



    the unnecessary horror that is ossec-control



    Which brings us to the complex stuff: that ossec-control script.



    It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.



    So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:




    1. systemd forks what it expects to be the service process.

    2. That's the shell, which forks ossec-control.

    3. That in turn forks between 4 and 10 grandchildren.

    4. The grandchildren all fork and exit in turn.

    5. The great-grandchildren all fork and exit in parallel.


    6. ossec-control exits.

    7. The first shell exits.

    8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.


    None of this horror is actually necessary under systemd at all. None of it.



    a systemd template service unit



    Instead, one writes a very simple template unit:




    [Unit]
    Description=The OSSEC HIDS %i server
    After=network.target

    [Service]
    Type=simple
    ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
    ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

    [Install]
    WantedBy=multi-user.target


    Save this this as /etc/systemd/system/ossec@.service.



    The various actual services are instantiations of this template, named:




    • ossec@dbd.service

    • ossec@agentlessd.service

    • ossec@csyslogd.service

    • ossec@execd.service

    • ossec@agentd.service

    • ossec@logcollector.service

    • ossec@syscheckd.service

    • ossec@maild.service

    • ossec@analysisd.service

    • ossec@remoted.service

    • ossec@monitord.service


    Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.


     systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord


    Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.



    By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.



    Further reading




    • Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.






    share|improve this answer





















    • 1





      Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

      – intelfx
      May 4 '15 at 18:45











    • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

      – Daniil Svetlov
      May 4 '15 at 22:14











    • What's the reason for using /usr/bin/env as a wrapper?

      – Marius Gedminas
      Mar 30 at 16:18
















    32














    readiness protocol mismatch



    As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.



    If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.



    You're doing exactly this.



    First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.



    In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.



    sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.



    the unnecessary horror that is ossec-control



    Which brings us to the complex stuff: that ossec-control script.



    It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.



    So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:




    1. systemd forks what it expects to be the service process.

    2. That's the shell, which forks ossec-control.

    3. That in turn forks between 4 and 10 grandchildren.

    4. The grandchildren all fork and exit in turn.

    5. The great-grandchildren all fork and exit in parallel.


    6. ossec-control exits.

    7. The first shell exits.

    8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.


    None of this horror is actually necessary under systemd at all. None of it.



    a systemd template service unit



    Instead, one writes a very simple template unit:




    [Unit]
    Description=The OSSEC HIDS %i server
    After=network.target

    [Service]
    Type=simple
    ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
    ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

    [Install]
    WantedBy=multi-user.target


    Save this this as /etc/systemd/system/ossec@.service.



    The various actual services are instantiations of this template, named:




    • ossec@dbd.service

    • ossec@agentlessd.service

    • ossec@csyslogd.service

    • ossec@execd.service

    • ossec@agentd.service

    • ossec@logcollector.service

    • ossec@syscheckd.service

    • ossec@maild.service

    • ossec@analysisd.service

    • ossec@remoted.service

    • ossec@monitord.service


    Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.


     systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord


    Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.



    By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.



    Further reading




    • Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.






    share|improve this answer





















    • 1





      Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

      – intelfx
      May 4 '15 at 18:45











    • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

      – Daniil Svetlov
      May 4 '15 at 22:14











    • What's the reason for using /usr/bin/env as a wrapper?

      – Marius Gedminas
      Mar 30 at 16:18














    32












    32








    32







    readiness protocol mismatch



    As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.



    If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.



    You're doing exactly this.



    First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.



    In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.



    sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.



    the unnecessary horror that is ossec-control



    Which brings us to the complex stuff: that ossec-control script.



    It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.



    So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:




    1. systemd forks what it expects to be the service process.

    2. That's the shell, which forks ossec-control.

    3. That in turn forks between 4 and 10 grandchildren.

    4. The grandchildren all fork and exit in turn.

    5. The great-grandchildren all fork and exit in parallel.


    6. ossec-control exits.

    7. The first shell exits.

    8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.


    None of this horror is actually necessary under systemd at all. None of it.



    a systemd template service unit



    Instead, one writes a very simple template unit:




    [Unit]
    Description=The OSSEC HIDS %i server
    After=network.target

    [Service]
    Type=simple
    ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
    ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

    [Install]
    WantedBy=multi-user.target


    Save this this as /etc/systemd/system/ossec@.service.



    The various actual services are instantiations of this template, named:




    • ossec@dbd.service

    • ossec@agentlessd.service

    • ossec@csyslogd.service

    • ossec@execd.service

    • ossec@agentd.service

    • ossec@logcollector.service

    • ossec@syscheckd.service

    • ossec@maild.service

    • ossec@analysisd.service

    • ossec@remoted.service

    • ossec@monitord.service


    Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.


     systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord


    Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.



    By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.



    Further reading




    • Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.






    share|improve this answer















    readiness protocol mismatch



    As Wieland implied, the Type of the service is important. That setting denotes what readiness protocol systemd expects the service to speak. A simple service is assumed to be immediately ready. A forking service is taken to be ready after its initial process forks a child and then exits. A dbus service is taken to be ready when a server appears on the Desktop Bus. And so forth.



    If you don't get the readiness protocol declared in the service unit to match what the service does, then things go awry. Readiness protocol mismatches cause services not to start correctly, or (more usually) to be (mis-)diagnosed by systemd as failing. When a service is seen as failing to start systemd ensures that every orphaned additional process of the service that might have been left running as part of the failure (from its point of view) is killed in order to bring the service properly back to the inactive state.



    You're doing exactly this.



    First of all, the simple stuff: sh -c doesn't match Type=simple or Type=forking.



    In the simple protocol, the initial process is taken to be the service process. But in fact a sh -c wrapper runs the actual service program as a child process. So MAINPID goes wrong and ExecReload stops working, for starters. When using Type=simple, one must either use sh -c 'exec …' or not use sh -c in the first place. The latter is more often the correct course than some people think.



    sh -c doesn't match Type=forking either. The readiness protocol for a forking service is quite specific. The initial process has to fork a child, and then exit. systemd applies a timeout to this protocol. If the initial process doesn't fork within the allotted time, it's a failure to become ready. If the initial process doesn't exit within the allotted time, that too is a failure.



    the unnecessary horror that is ossec-control



    Which brings us to the complex stuff: that ossec-control script.



    It turns out that it's a System 5 rc script that forks off between 4 and 10 processes, which themselves in their turn fork and exit too. It's one of those System 5 rc scripts that attempts to manage a whole set of server processes in one single script, with for loops, race conditions, arbitrary sleeps to try to avoid them, failure modes that can choke the system in a half-started state, and all of the other horrors that got people inventing things like the AIX System Resource Controller and daemontools two decades ago. And let's not forget the hidden shell script in a binary directory that it rewrites on the fly, to implement idiosyncratic enable and disable verbs.



    So when you /bin/sh -c '/var/ossec/bin/ossec-control start' what happens is that:




    1. systemd forks what it expects to be the service process.

    2. That's the shell, which forks ossec-control.

    3. That in turn forks between 4 and 10 grandchildren.

    4. The grandchildren all fork and exit in turn.

    5. The great-grandchildren all fork and exit in parallel.


    6. ossec-control exits.

    7. The first shell exits.

    8. The service processes were the great-great-grandchildren, but because this way of working matches neither the forking nor the simple readiness protocol, systemd considers the service as a whole to have failed and shuts it back down.


    None of this horror is actually necessary under systemd at all. None of it.



    a systemd template service unit



    Instead, one writes a very simple template unit:




    [Unit]
    Description=The OSSEC HIDS %i server
    After=network.target

    [Service]
    Type=simple
    ExecStartPre=/usr/bin/env /var/ossec/bin/%p-%i -t
    ExecStart=/usr/bin/env /var/ossec/bin/%p-%i -f

    [Install]
    WantedBy=multi-user.target


    Save this this as /etc/systemd/system/ossec@.service.



    The various actual services are instantiations of this template, named:




    • ossec@dbd.service

    • ossec@agentlessd.service

    • ossec@csyslogd.service

    • ossec@execd.service

    • ossec@agentd.service

    • ossec@logcollector.service

    • ossec@syscheckd.service

    • ossec@maild.service

    • ossec@analysisd.service

    • ossec@remoted.service

    • ossec@monitord.service


    Then enable and disable function comes straight from the service management system (with RedHat bug 752774 fixed), with no need for hidden shell scripts.


     systemctl enable ossec@dbd ossec@agentlessd ossec@csyslogd ossec@maild ossec@execd ossec@analysisd ossec@logcollector ossec@remoted ossec@syscheckd ossec@monitord


    Moreover, systemd gets to know about, and to track, each actual service directly. It can filter their logs with journalctl -u. It can know when an individual service has failed. It knows what services are supposed to be enabled and running.



    By the way: Type=simple and the -f option are as right here as they are in many other cases. Very few services in the wild actually signal their readiness by dint of the exit, and these here are not such cases either. But that's what the forking type means. Services in the wild in the main just fork and exit because of some mistaken received wisdom notion that that's what dæmons are supposed to do. In fact, it's not. It hasn't been since the 1990s. It's time to catch up.



    Further reading




    • Jonathan de Boyne Pollard (2015). Readiness protocol problems with Unix dæmons. Frequently Given Answers.







    share|improve this answer














    share|improve this answer



    share|improve this answer








    edited Sep 26 '17 at 2:55

























    answered May 4 '15 at 17:55









    JdeBPJdeBP

    39.5k482191




    39.5k482191








    • 1





      Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

      – intelfx
      May 4 '15 at 18:45











    • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

      – Daniil Svetlov
      May 4 '15 at 22:14











    • What's the reason for using /usr/bin/env as a wrapper?

      – Marius Gedminas
      Mar 30 at 16:18














    • 1





      Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

      – intelfx
      May 4 '15 at 18:45











    • @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

      – Daniil Svetlov
      May 4 '15 at 22:14











    • What's the reason for using /usr/bin/env as a wrapper?

      – Marius Gedminas
      Mar 30 at 16:18








    1




    1





    Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

    – intelfx
    May 4 '15 at 18:45





    Very detailed answer! I'd also suggest to create a "grouping" target, say, ossec.target, which Requires= all the needed instances, and then set PartOf=ossec.target in ossec@.service. This will allow to start and stop ossec by starting and stopping ossec.target.

    – intelfx
    May 4 '15 at 18:45













    @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

    – Daniil Svetlov
    May 4 '15 at 22:14





    @JdeBP, wow! Thank a lot for such kind of detailed answer. Hope I'll make this unit and write here about results. I was though, that I'll be easier. But you are right, ossec-controll is a init hell.

    – Daniil Svetlov
    May 4 '15 at 22:14













    What's the reason for using /usr/bin/env as a wrapper?

    – Marius Gedminas
    Mar 30 at 16:18





    What's the reason for using /usr/bin/env as a wrapper?

    – Marius Gedminas
    Mar 30 at 16:18













    0














    Keep Type=forking and give a pid file location if start service/app is maintaining any pid.



    [Unit]

    Description="Run app on boot"

    After=network.target syslog.target auditd.service


    [Service]

    Type=forking

    PIDFile=/var/run/apache2/apache2.pid

    ExecStart=/etc/init.d/apache2 start

    ExecStop=/etc/init.d/apache2 stop

    StandardOutput=syslog

    StandardError=syslog

    Restart=on-failure

    SyslogIdentifier=webappslog


    [Install]

    WantedBy=multi-user.target

    Alias=webapps






    share|improve this answer








    New contributor



    Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
    Check out our Code of Conduct.
























      0














      Keep Type=forking and give a pid file location if start service/app is maintaining any pid.



      [Unit]

      Description="Run app on boot"

      After=network.target syslog.target auditd.service


      [Service]

      Type=forking

      PIDFile=/var/run/apache2/apache2.pid

      ExecStart=/etc/init.d/apache2 start

      ExecStop=/etc/init.d/apache2 stop

      StandardOutput=syslog

      StandardError=syslog

      Restart=on-failure

      SyslogIdentifier=webappslog


      [Install]

      WantedBy=multi-user.target

      Alias=webapps






      share|improve this answer








      New contributor



      Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
      Check out our Code of Conduct.






















        0












        0








        0







        Keep Type=forking and give a pid file location if start service/app is maintaining any pid.



        [Unit]

        Description="Run app on boot"

        After=network.target syslog.target auditd.service


        [Service]

        Type=forking

        PIDFile=/var/run/apache2/apache2.pid

        ExecStart=/etc/init.d/apache2 start

        ExecStop=/etc/init.d/apache2 stop

        StandardOutput=syslog

        StandardError=syslog

        Restart=on-failure

        SyslogIdentifier=webappslog


        [Install]

        WantedBy=multi-user.target

        Alias=webapps






        share|improve this answer








        New contributor



        Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.









        Keep Type=forking and give a pid file location if start service/app is maintaining any pid.



        [Unit]

        Description="Run app on boot"

        After=network.target syslog.target auditd.service


        [Service]

        Type=forking

        PIDFile=/var/run/apache2/apache2.pid

        ExecStart=/etc/init.d/apache2 start

        ExecStop=/etc/init.d/apache2 stop

        StandardOutput=syslog

        StandardError=syslog

        Restart=on-failure

        SyslogIdentifier=webappslog


        [Install]

        WantedBy=multi-user.target

        Alias=webapps







        share|improve this answer








        New contributor



        Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.








        share|improve this answer



        share|improve this answer






        New contributor



        Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.








        answered 37 mins ago









        RaushanRaushan

        1




        1




        New contributor



        Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.




        New contributor




        Raushan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
        Check out our Code of Conduct.

























            -4














            Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.



            JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.






            share|improve this answer
























            • What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

              – JoostM
              Nov 11 '18 at 10:17


















            -4














            Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.



            JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.






            share|improve this answer
























            • What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

              – JoostM
              Nov 11 '18 at 10:17
















            -4












            -4








            -4







            Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.



            JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.






            share|improve this answer













            Note that systemd's daemon model is simplistic and incompatible with many existing daemons which do multiple forking, exec'ing and setuid'ing. Most common are the daemons which start as root to set things up and then switch to a less privileged UID for routine operation. e.g. Pid file initialization is one thing which fails under systemd due to privilege problems. There are workarounds (not fixes) but it's badly documented.



            JdeBP's explanation is welcome but incomplete and his claim that it's all ossec-control's fault is simply not true. Even quite trivial stuff is problematic e.g. getting untruncated log lines to debug problems or meaningful error messages from systemd itself when it kills processes.







            share|improve this answer












            share|improve this answer



            share|improve this answer










            answered Sep 20 '16 at 7:07









            JohnJohn

            11




            11













            • What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

              – JoostM
              Nov 11 '18 at 10:17





















            • What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

              – JoostM
              Nov 11 '18 at 10:17



















            What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

            – JoostM
            Nov 11 '18 at 10:17







            What use are PID files, anyway? If one exists for a given service, there may or may not be an actual process with that PID, and when a process with the right PID does exist, it may or may not actually be the expected service.

            – JoostM
            Nov 11 '18 at 10:17




















            draft saved

            draft discarded




















































            Thanks for contributing an answer to Unix & Linux Stack Exchange!


            • Please be sure to answer the question. Provide details and share your research!

            But avoid



            • Asking for help, clarification, or responding to other answers.

            • Making statements based on opinion; back them up with references or personal experience.


            To learn more, see our tips on writing great answers.




            draft saved


            draft discarded














            StackExchange.ready(
            function () {
            StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2funix.stackexchange.com%2fquestions%2f200280%2fsystemd-kills-service-immediately-after-start%23new-answer', 'question_page');
            }
            );

            Post as a guest















            Required, but never shown





















































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown

































            Required, but never shown














            Required, but never shown












            Required, but never shown







            Required, but never shown







            Popular posts from this blog

            Taj Mahal Inhaltsverzeichnis Aufbau | Geschichte | 350-Jahr-Feier | Heutige Bedeutung | Siehe auch |...

            Baia Sprie Cuprins Etimologie | Istorie | Demografie | Politică și administrație | Arii naturale...

            Nicolae Petrescu-Găină Cuprins Biografie | Opera | In memoriam | Varia | Controverse, incertitudini...