[Bug]: Service re-run terminates despite available fleet capacity.

[Bihan]

Steps to reproduce

Configs:

# my_cpu_fleet.yml
type: fleet
name: cpu-default

nodes: 0..8

resources:
 cpu: 2

# simple-service-replicas.yml
type: service
name: simple-service-replicas
https: false
python: 3.12


commands:
 - echo "Group default - Version 1" > /tmp/version.txt
 - python3 -m http.server 8000

port: 8000

resources:
 cpu: 2

replicas: 5

Step1: Create Fleet: dstack apply -f my_cpu_fleet.yml

Step2: Apply Service Config dstack apply -f simple-service-replicas.yml

The first run works as expected

dstack ps
 NAME BACKEND GPU PRICE STATUS SUBMITTED 
 simple-service-replicas - - running 5 mins ago 
 replica=0 aws (us-east-2) - $0.0832 running 5 mins ago 
 replica=1 aws (us-east-2) - $0.0832 running 5 mins ago 
 replica=2 aws (us-east-2) - $0.0832 running 5 mins ago 
 replica=3 aws (us-east-2) - $0.0832 running 5 mins ago 
 replica=4 aws (us-east-2) - $0.0832 running 5 mins ago 

dstack fleet
 FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
 default - - - - - 4 days ago 
 cpu-default 0 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 4 mins ago 
 1 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 4 mins ago 
 2 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 4 mins ago 
 3 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 4 mins ago 
 4 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 4 mins ago 

Step3: Stop the run. fleet instances are idle as expected.

FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
default - - - - - 4 days ago 
cpu-default 0 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 6 mins ago 
 1 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 5 mins ago 
 2 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 5 mins ago 
 3 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 5 mins ago 
 4 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 5 mins ago 

Step4: Once Again apply: dstack apply -f simple-service-replicas.yml

dstack ps
 NAME BACKEND GPU PRICE STATUS SUBMITTED 
 simple-service-replicas - - running 56 sec ago 
 replica=0 aws (us-east-2) - $0.0832 running 55 sec ago 
 replica=1 aws (us-east-2) - $0.0832 pulling 55 sec ago 
 replica=2 aws (us-east-2) - $0.0832 pulling 55 sec ago 
 replica=3 aws (us-east-2) - $0.0832 pulling 55 sec ago 
 replica=4 aws (us-east-2) - $0.0832 running 55 sec ago 

All the fleet instances are expected to be busy when replica's are pulling/running, but some are idle as below:

dstack fleet
 FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
 default - - - - - 4 days ago 
 cpu-default 0 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 10 mins ago 
 1 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 2 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 3 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 4 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 10 mins ago 

Step5: Check the run after a while

dstack ps
 NAME BACKEND GPU PRICE STATUS SUBMITTED 
 simple-service-replicas - - terminating 3 mins ago 
 replica=0 aws (us-east-2) - $0.0832 running 3 mins ago 
 replica=1 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=2 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=3 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=4 aws (us-east-2) - $0.0832 running 3 mins ago

The run gets terminated.

Actual behaviour

The run gets terminated on re-run even when fleet has idle instances.

dstack fleet
 FLEET INSTANCE BACKEND RESOURCES PRICE STATUS CREATED 
 default - - - - - 4 days ago 
 cpu-default 0 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 10 mins ago 
 1 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 2 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 3 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 idle 10 mins ago 
 4 aws (us-east-2) cpu=2 mem=8GB disk=100GB $0.0832 busy 10 mins ago 

dstack ps
 NAME BACKEND GPU PRICE STATUS SUBMITTED 
 simple-service-replicas - - terminating 3 mins ago 
 replica=0 aws (us-east-2) - $0.0832 running 3 mins ago 
 replica=1 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=2 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=3 aws (us-east-2) - $0.0832 terminating 3 mins ago 
 replica=4 aws (us-east-2) - $0.0832 running 3 mins ago

Expected behaviour

The re-run should not be terminated and idle fleet instances should be utilized.

dstack version

master (commit: b2be6a7)

Server logs

Additional information

server_logs_fleet_issue.txt

[jvstme]

The problem is that multiple jobs are assigned to the same instance, even though it only has 1 block.

INFO dstack._internal.server.services.events:178 Emitting event: Job assigned to instance. Instance status: BUSY (1/1 blocks busy). Event targets: EventTargetType.JOB(826505)simple-service-replicas-0-0, EventTargetType.INSTANCE(3ab278)cpu-default-4. Actor: system
INFO dstack._internal.server.services.events:178 Emitting event: Job assigned to instance. Instance status: BUSY (1/1 blocks busy). Event targets: EventTargetType.JOB(b3cba7)simple-service-replicas-0-3, EventTargetType.INSTANCE(5bc2bc)cpu-default-0. Actor: system
INFO dstack._internal.server.services.events:178 Emitting event: Job assigned to instance. Instance status: BUSY (1/1 blocks busy). Event targets: EventTargetType.JOB(23d3e9)simple-service-replicas-0-2, EventTargetType.INSTANCE(5bc2bc)cpu-default-0. Actor: system
INFO dstack._internal.server.services.events:178 Emitting event: Job assigned to instance. Instance status: BUSY (1/1 blocks busy). Event targets: EventTargetType.JOB(1912d0)simple-service-replicas-0-4, EventTargetType.INSTANCE(5bc2bc)cpu-default-0. Actor: system
INFO dstack._internal.server.services.events:178 Emitting event: Job assigned to instance. Instance status: BUSY (1/1 blocks busy). Event targets: EventTargetType.JOB(6748fe)simple-service-replicas-0-1, EventTargetType.INSTANCE(5bc2bc)cpu-default-0. Actor: system

This then leads to 409 errors when trying to start the jobs on the instance, which eventually causes jobs to fail.

DEBUG dstack._internal.server.services.runner.ssh:105 Cannot connect to 3.23.115.47's API: 409 Client Error: Conflict for url: http://localhost:51662/api/submit 
WARNING dstack._internal.server.background.tasks.process_running_jobs:396 job(b3cba7)simple-service-replicas-0-3: is unreachable, waiting for the instance to become reachable again, age=0:00:49.337176

I reproduced the issue and will investigate further.

Another problem is that there was a gateway error when registering one of the replicas.

[18:48:18] INFO dstack._internal.server.services.events:178 Emitting event: Job status changed PULLING -> RUNNING. Event targets: EventTargetType.JOB(23d3e9)simple-service-replicas-0-2. Actor: system
 DEBUG dstack._internal.server.services.services:187 job(23d3e9)simple-service-replicas-0-2: registering replica for service c3339df5b2564faf8761b476fc915c92
 DEBUG dstack._internal.server.services.services:197 Gateway request failed
 Traceback (most recent call last):
 [... cut for brevity ...]
 File "/Users/bihan/Projects/dstack-bihan/dstack/src/dstack/_internal/server/services/gateways/client.py", line 126, in register_replica
 resp = await self._client.post(
 ^^^^^^^^^^^^^^^^^^^^^^^^
 [... cut for brevity ...]
 File "/Users/bihan/Projects/dstack-bihan/dstack/.venv/lib/python3.11/site-packages/httpx/_transports/default.py", line 118, in map_httpcore_exceptions
 raise mapped_exc(message) from exc
 httpx.RemoteProtocolError: Server disconnected without sending a response.
 WARNING dstack._internal.server.background.tasks.process_running_jobs:858 job(23d3e9)simple-service-replicas-0-2: failed to register service replica: RemoteProtocolError('Server disconnected without sending a response.')
 INFO dstack._internal.server.services.events:178 Emitting event: Job status changed RUNNING -> TERMINATING. Termination reason: GATEWAY_ERROR (Failed to register service replica). Event targets: EventTargetType.JOB(23d3e9)simple-service-replicas-0-2. Actor: system

I couldn't reproduce this one and I'm not sure if it's related. @Bihan, could you also share the gateway logs for the moment when this exception occurred?

[Bihan]

gateway_logs.txt
@jvstme Here is the gateway logs around that time

[jvstme]

These are the gateway logs about replica 23d3e9 that the server failed to register:

Dec 19 13:03:19 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:19,068 - dstack._internal.proxy.gateway.services.registry - DEBUG - Registering replica 82650535>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,460 - dstack._internal.proxy.gateway.services.nginx - DEBUG - Registering service domain simp>
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/cp -p /tmp/tmpw2kl4mf9 /etc/nginx/sites-enabl>
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: pam_unix(sudo:session): session closed for user root
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/systemctl reload nginx.service
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: pam_unix(sudo:session): session closed for user root
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,552 - dstack._internal.proxy.gateway.services.nginx - INFO - Registered service domain simple>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,553 - dstack._internal.proxy.gateway.services.registry - INFO - Replica 826505350d1a47b5aa1a4>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: INFO: 127.0.0.1:52410 - "POST /api/registry/main/services/simple-service-replicas/replicas/register HTTP/1.1">

Nothing special here. If this replica registration error doesn't occur often, it may be due to some server-to-gateway communication issue.

So the main problem I see is multiple jobs being assigned to the same instance, which causes those jobs to fail. Apparently, the instance isn’t locked before job assignment, resulting in a race condition. We'll need to find a way to fix it

[Bihan]

These are the gateway logs about replica 23d3e9 that the server failed to register:

Dec 19 13:03:19 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:19,068 - dstack._internal.proxy.gateway.services.registry - DEBUG - Registering replica 82650535>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,460 - dstack._internal.proxy.gateway.services.nginx - DEBUG - Registering service domain simp>
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/cp -p /tmp/tmpw2kl4mf9 /etc/nginx/sites-enabl>
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Dec 19 13:03:20 ip-172-31-21-247 sudo[29627]: pam_unix(sudo:session): session closed for user root
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: ubuntu : PWD=/home/ubuntu/dstack ; USER=root ; COMMAND=/usr/bin/systemctl reload nginx.service
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=1000)
Dec 19 13:03:20 ip-172-31-21-247 sudo[29629]: pam_unix(sudo:session): session closed for user root
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,552 - dstack._internal.proxy.gateway.services.nginx - INFO - Registered service domain simple>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: 2025-12-19 13:03:20,553 - dstack._internal.proxy.gateway.services.registry - INFO - Replica 826505350d1a47b5aa1a4>
Dec 19 13:03:20 ip-172-31-21-247 sh[29500]: INFO: 127.0.0.1:52410 - "POST /api/registry/main/services/simple-service-replicas/replicas/register HTTP/1.1">

Nothing special here. If this replica registration error doesn't occur often, it may be due to some server-to-gateway communication issue.

So the main problem I see is multiple jobs being assigned to the same instance, which causes those jobs to fail. Apparently, the instance isn’t locked before job assignment, resulting in a race condition. We'll need to find a way to fix it

Yes, the replica registration problem does not occur often. That was the only time I saw it. The main problem is the race condition.