Understanding the Problem

What Is the Activating (Start) Mode?

When a service is in the activating state, systemd has started the process but is waiting for a notification or event to confirm readiness. This is particularly common with services using Type=notify or Type=dbus. If the notification never arrives, the service hangs in this state indefinitely or until the timeout is reached.

Symptoms in Production Environments

  • Service dependencies never load (e.g., web server waiting on a database).
  • System boot sequence stalls.
  • systemctl status shows "activating (start)" without logs advancing.
  • CI/CD runners hang during provisioning stages.

Root Causes

  • Misconfigured unit file: Using Type=notify or Type=dbus without the service emitting readiness signals.
  • Socket-based activation issues: When socket units are not correctly linked to services.
  • TimeoutStartSec too high or missing: Leading to long or infinite waits.
  • Service code hangs or fails silently: Underlying binary fails before readiness notification.
  • Missing permissions or AppArmor restrictions: Preventing service from executing required operations.

Diagnostic Steps

1. Check Unit File Configuration

systemctl cat my-service.service

Focus on these directives:

  • Type=
  • ExecStart=
  • TimeoutStartSec=

If Type=notify is set but the binary doesn’t use sd_notify, change to Type=simple or adjust the service code.

2. Review Journal Logs

journalctl -u my-service.service --no-pager --since "5m ago"

Look for signs of hangs, permission denials, or failed readiness checks.

3. Temporarily Modify Unit for Debugging

Add StandardOutput=journal+console to improve log visibility:

[Service]
ExecStart=/usr/bin/my-binary
StandardOutput=journal+console
TimeoutStartSec=30

Reload systemd and restart:

systemctl daemon-reexec
systemctl daemon-reload
systemctl restart my-service

4. Use strace for Deeper Inspection

If the binary is hanging, attach strace:

pidof my-binary
strace -p <PID>

This reveals system calls that might be blocking (e.g., futex, socket, read).

5. Analyze Dependencies

systemctl list-dependencies my-service.service

Check for circular dependencies or upstream units that are also stuck.

Common Fixes

  • Set Type=simple unless notification is explicitly implemented.
  • Define TimeoutStartSec=30 or another reasonable value to avoid indefinite activation.
  • Ensure service binary is not waiting on stdin/stdout in systemd context.
  • Use Restart=on-failure to recover from non-zero exit codes.
  • Audit SELinux/AppArmor profiles if access denials are suspected.

Best Practices for Preventing Future Occurrences

  • Use systemd-analyze verify to validate unit files before deployment.
  • Prefer stateless service design—ensure services do not require user interaction at startup.
  • Use dedicated logging instead of silent failures inside scripts or custom daemons.
  • Document systemd dependencies using Wants= and After= judiciously.
  • In CI/CD pipelines, add validation steps to confirm services transition to active.

Conclusion

Services stuck in "activating (start)" state on Ubuntu can grind operations to a halt, especially in automated and containerized environments. The key is understanding how systemd expects services to behave during startup, particularly around readiness signals and dependency management. By applying structured diagnostics and enforcing consistent unit configurations, you can dramatically reduce the risk of services hanging during critical system operations.

FAQs

1. What causes systemd to expect notifications from a service?

If Type=notify is used in the unit file, systemd waits for a readiness signal (sd_notify). Without it, the service stays in activating state.

2. How do I know if my service sends sd_notify signals?

Check the source code for sd_notify() calls or verify with strace whether it sends to the NOTIFY_SOCKET.

3. Can I override a broken systemd unit from a package?

Yes. Create a drop-in config in /etc/systemd/system/<unit>.d/override.conf and override problematic directives like Type= or TimeoutStartSec=.

4. Does using Type=simple introduce risks?

Not usually. It tells systemd the service is running after ExecStart launches. It’s safe for services that don't need to signal readiness explicitly.

5. How can I debug this during boot?

Use the recovery shell, access logs via journalctl --boot, or temporarily disable DefaultDependencies=no for debugging critical units.