AWS - Blockstack Gaia Service Dependency failed

My AWS EC2 instance keeps timing out. I have opened up the necessary ports on AWS now i’m attempting the below located on blockstack to restart the certs since the connection keeps timing out when i try to access it. Below is the issue i’m facing after running the command to restart gaia.service

-MacBook-Air GaiaAws % ssh -t -i gaiaKeyPair.pem -A [email protected] “sudo systemctl restart gaia.service”

A dependency job for gaia.service failed. See ‘journalctl -xe’ for details.

Connection to 100.26.156.236 closed.

some more info here is required.
can you ssh to your host and run something like systemctl status gaia.service
or
journalctl -xe -u gaia.service?
i suspect it’s a dependent service that is failing, but it’s unclear which one

The output of running.status gaia.service is below

**[email protected]** **~ $** sudo systemctl status gaia.service

● gaia.service - Gaia

Loaded: loaded (/etc/systemd/system/gaia.service; enabled; vendor preset: enabled)

Active: inactive (dead)

Jun 18 12:04:06 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

Jun 18 12:04:06 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 18 16:20:52 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

Jun 18 16:20:52 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 18 17:16:13 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

Jun 18 17:16:13 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 19 00:07:42 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

Jun 19 00:07:42 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 19 12:03:57 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

Jun 19 12:03:57 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**
-- 

-- The job identifier is 82119 and the job result is dependency.

Jun 18 16:20:52 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 18 17:16:13 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

-- Subject: A start job for unit gaia.service has failed

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit gaia.service has finished with a failure.

-- 

-- The job identifier is 89500 and the job result is dependency.

Jun 18 17:16:13 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 19 00:07:42 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

-- Subject: A start job for unit gaia.service has failed

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit gaia.service has finished with a failure.

-- 

-- The job identifier is 142948 and the job result is dependency.

Jun 19 00:07:42 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

Jun 19 12:03:57 ip-172-31-61-77 systemd[1]: **Dependency failed for Gaia.**

-- Subject: A start job for unit gaia.service has failed

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit gaia.service has finished with a failure.

-- 

-- The job identifier is 274610 and the job result is dependency.

Jun 19 12:03:57 ip-172-31-61-77 systemd[1]: **gaia.service: Job gaia.service/start failed with result 'dependency'.**

try the same commands on the following services:
check-dns.service
letsencrypt-init.service
the issue likely lies with one of those

Running

watch -n 2 -t -g -x host maryhub.ga

blockstackgaia.ga has address 100.26.238.19 which is correct

my userdata

{
“ignition”: { “version”: “2.2.0” },
“storage”: {
“files”: [{
“filesystem”: “root”,
“path”: “/etc/environment”,
“mode”: 420,
“contents”: {
“source”: “data:application/octet-stream,API_KEY%3Dhubba%0ADOMAIN%3Dblockstackgaia.ga%0ASTAGING%3D0”
}
}]
}
}

check-dns.service below

[email protected] ~ $ sudo systemctl status check-dns.service
● check-dns.service - Check DNS Record Service
   Loaded: loaded (/etc/systemd/system/check-dns.service; disabled; vendor preset: disabled)
   Active: activating (start) since Tue 2020-06-23 12:58:31 UTC; 10min ago
 Main PID: 13053 (bash)
    Tasks: 2 (limit: 15574)
   Memory: 1.1M
   CGroup: /system.slice/check-dns.service
           ├─13053 /bin/bash /gaia/scripts/check-dns.sh
           └─13718 sleep 80

Jun 23 13:00:16 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 35s....
Jun 23 13:00:51 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 40s....
Jun 23 13:01:31 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 45s....
Jun 23 13:02:16 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 50s....
Jun 23 13:03:06 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 55s....
Jun 23 13:04:01 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 60s....
Jun 23 13:05:01 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 65s....
Jun 23 13:06:06 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 70s....
Jun 23 13:07:16 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 75s....
Jun 23 13:08:31 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 80s....

**[email protected]** **~ $** sudo journalctl -xe -u check-dns.service

-- An ExecStart= process belonging to unit check-dns.service has exited.

-- 

-- The process' exit code is 'exited' and its exit status is 1.

Jun 23 13:09:51 ip-172-31-61-77 systemd[1]: **check-dns.service: Failed with result 'exit-code'.**

-- Subject: Unit failed

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- The unit check-dns.service has entered the 'failed' state with result 'exit-code'.

Jun 23 13:09:51 ip-172-31-61-77 systemd[1]: **Failed to start Check DNS Record Service.**

-- Subject: A start job for unit check-dns.service has failed

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit check-dns.service has finished with a failure.

-- 

-- The job identifier is 139647 and the job result is failed.

Jun 23 13:09:51 ip-172-31-61-77 systemd[1]: Starting Check DNS Record Service...

-- Subject: A start job for unit check-dns.service has begun execution

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit check-dns.service has begun execution.

-- 

-- The job identifier is 144803.

Jun 23 13:09:51 ip-172-31-61-77 bash[13745]: Current epoch: 1592917791

Jun 23 13:09:51 ip-172-31-61-77 bash[13745]: End epoch: 1592918392

Jun 23 13:09:51 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 5s....

Jun 23 13:09:56 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 10s....

Jun 23 13:10:06 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 15s....

Jun 23 13:10:21 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 20s....

Jun 23 13:10:41 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 25s....

Jun 23 13:11:06 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 30s....

Jun 23 13:11:36 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 35s....

Jun 23 13:12:11 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 40s....

Jun 23 13:12:51 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 45s....

Jun 23 13:13:36 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 50s....

Jun 23 13:14:26 ip-172-31-61-77 bash[13745]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 55s....

letsencrypt-init.service below

**[email protected]** **~ $** sudo systemctl status letsencrypt-init.service

● letsencrypt-init.service - LetsEncrypt Init Service

Loaded: loaded (/etc/systemd/system/letsencrypt-init.service; disabled; vendor preset: disabled)

Active: inactive (dead) since Tue 2020-06-23 02:59:36 UTC; 10h ago

Condition: start **condition failed** at Tue 2020-06-23 13:10:25 UTC; 47s ago

└─ ConditionPathExists=!/tmp/letsencrypt.init was not met

Process: 631 ExecStart=/bin/bash -x /gaia/scripts/letsencrypt-init.sh (code=exited, status=0/SUCCESS)

 Main PID: 631 (code=exited, status=0/SUCCESS)

Jun 23 13:01:14 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:02:17 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:03:17 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:04:18 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:05:18 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:06:18 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:07:22 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:08:22 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:09:23 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 13:10:25 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

**[email protected]** **~ $** sudo journalctl -xe -u letsencrypt-init.service

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit letsencrypt-init.service has finished successfully.

-- 

-- The job identifier is 144289.

Jun 23 13:09:23 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

-- Subject: A start job for unit letsencrypt-init.service has finished successfully

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit letsencrypt-init.service has finished successfully.

-- 

-- The job identifier is 144731.

Jun 23 13:10:25 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

-- Subject: A start job for unit letsencrypt-init.service has finished successfully

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit letsencrypt-init.service has finished successfully.

-- 

-- The job identifier is 145541.

Jun 23 13:11:26 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

-- Subject: A start job for unit letsencrypt-init.service has finished successfully

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit letsencrypt-init.service has finished successfully.

-- 

-- The job identifier is 145909.

Jun 23 13:12:27 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

-- Subject: A start job for unit letsencrypt-init.service has finished successfully

-- Defined-By: systemd

-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- 

-- A start job for unit letsencrypt-init.service has finished successfully.

-- 

-- The job identifier is 146277.

the issue here is: NAME_OF_DOMAIN.
did you add the user-data after the first boot? the way ignition works is that it’s a one-time change (changing the userdata later is not generally supported).

the file that looks like it might be the issue here is /etc/environment, which most units use for global variables like DOMAIN.
for example, this line: Jun 23 13:00:16 ip-172-31-61-77 bash[13053]: [ NAME_OF_DOMAIN ] Record () doesn't match public IP (100.26.238.19) - sleeping for 35s....

i would expect this output to be: Jun 23 13:00:16 ip-172-31-61-77 bash[13053]: [ blockstackgaia.ga ] Record () doesn't match public IP (100.26.238.19) - sleeping for 35s....

is from the script /gaia/scripts/check-dns.sh, line 22: https://github.com/blockstackpbc/gaia-docker/blob/master/scripts/check-dns.sh#L22

which in turn is loaded into any unit that needs access to those values: https://github.com/blockstackpbc/gaia-docker/blob/master/unit-files/check-dns.service#L12

if you update the ignition user-data after the first boot, that file isn’t updated.

at this point though, it’s probably easiest to just vi /etc/environment and make the changes by hand.

if this works, i’d appreciate a quick walkthrough of how you stood this up and what steps you took - it sounds like there might be some confusion in the docs that should be addressed.

Thanks, updated the environment. It now shows the expected log you was hoping for but didn’t fix the issue :\

When i try to restart gaia service i’m prompted still with the dependency error. And yes this instance was after first boot. I created a new instance to test your theory. The environment did pick up the info but i still received the same issue as below

[email protected] ~ % nmap -Pn 100.26.238.19 -p 22,80,443
Starting Nmap 7.80 ( https://nmap.org ) at 2020-06-23 16:02 EDT
Nmap scan report for ec2-100-26-238-19.compute-1.amazonaws.com (100.26.238.19)
Host is up (0.037s latency).

PORT    STATE  SERVICE
22/tcp  open   ssh
80/tcp  closed http
443/tcp closed https

[email protected] ~ $ sudo systemctl status check-dns.service

● check-dns.service - Check DNS Record Service

Loaded: loaded (/etc/systemd/system/check-dns.service; disabled; vendor preset: disabled)

Active: inactive (dead) since Tue 2020-06-23 19:52:43 UTC; 4min 46s ago

Condition: start condition failed at Tue 2020-06-23 19:56:55 UTC; 35s ago

└─ ConditionPathExists=!/tmp/dns_checked was not met

Process: 25908 ExecStart=/bin/bash /gaia/scripts/check-dns.sh (code=exited, status=0/SUCCESS)

Main PID: 25908 (code=exited, status=0/SUCCESS)

Jun 23 19:52:43 ip-172-31-61-77 systemd[1]: Starting Check DNS Record Service…

Jun 23 19:52:43 ip-172-31-61-77 bash[25908]: Current epoch: 1592941963

Jun 23 19:52:43 ip-172-31-61-77 bash[25908]: End epoch: 1592942564

Jun 23 19:52:43 ip-172-31-61-77 bash[25908]: [ blockstackgaia.ga IN A 100.26.238.19 ] Matched 100.26.238.19

Jun 23 19:52:43 ip-172-31-61-77 systemd[1]: check-dns.service: Succeeded.

Jun 23 19:52:43 ip-172-31-61-77 systemd[1]: Started Check DNS Record Service.

Jun 23 19:53:48 ip-172-31-61-77 systemd[1]: Condition check resulted in Check DNS Record Service being skipped.

Jun 23 19:54:48 ip-172-31-61-77 systemd[1]: Condition check resulted in Check DNS Record Service being skipped.

Jun 23 19:55:54 ip-172-31-61-77 systemd[1]: Condition check resulted in Check DNS Record Service being skipped.

Jun 23 19:56:55 ip-172-31-61-77 systemd[1]: Condition check resulted in Check DNS Record Service being skipped.


[email protected] ~ $ sudo systemctl status letsencrypt-init.service

● letsencrypt-init.service - LetsEncrypt Init Service

Loaded: loaded (/etc/systemd/system/letsencrypt-init.service; disabled; vendor preset: disabled)

Active: inactive (dead) since Tue 2020-06-23 02:59:36 UTC; 17h ago

Condition: start condition failed at Tue 2020-06-23 20:00:02 UTC; 2s ago

└─ ConditionPathExists=!/tmp/letsencrypt.init was not met

Process: 631 ExecStart=/bin/bash -x /gaia/scripts/letsencrypt-init.sh (code=exited, status=0/SUCCESS)

Main PID: 631 (code=exited, status=0/SUCCESS)

Jun 23 19:50:29 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:51:30 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:52:31 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:53:33 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:54:34 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:55:54 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:56:55 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:57:58 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 19:59:00 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

Jun 23 20:00:02 ip-172-31-61-77 systemd[1]: Condition check resulted in LetsEncrypt Init Service being skipped.

on a new VM, the file /tmp/letsencrypt.init should not exist.
rm /tmp/letsencrypt.init and sudo systemctl restart letsencrypt-init.service

this whole process likely needs an update i suspect

Thanks. I’m no longer receiving the

A dependency job for gaia.service failed. See ‘journalctl -xe’ for details.

After deleting the file you suggested. I was able to continue on with the EC2 tutorial on blockstack without receiving any errors. The only “off” thing is i didn’t receive the “connection closed”. My connection is still timing out though with the tcp’s port 80,443 still being closed :\

**[email protected]** **/ $** sudo rm /tmp/letsencrypt.init

**[email protected]** **/ $** sudo systemctl restart letsencrypt-init.service

**[email protected]** **/ $** sudo systemctl restart gaia.service

**[email protected]** **/ $** sudo systemctl restart reset-ssl-certs.service

hmmm. quite odd. it’s been a while since i’ve walked through this process myself.
ports 80 & 443 are opened by nginx, which has it’s own unit file.

systemctl restart nginx
journalctl -xe -u nginx

I’ll have to take a closer look at this soon though, something seems off since you shouldn’t have seen the issues you did.

So after removing the /tmp and restarting the lets-encrypt file. I’m now seeing this error when ssh into my console

[email protected] AWS % ssh -i "blockstackPair.pem" [email protected]

Last login: Tue Jun 23 20:27:10 UTC 2020 from 75.97.131.83 on pts/0

Container Linux by CoreOS stable (2135.6.0)

*

* https://github.com/blockstack/gaia

*

Update Strategy: No Reboots

Failed Units: 2

letsencrypt.service

nginx.service

**[email protected]** **~ $**

**[email protected]** **~ $** journalctl -xe -u nginx

No journal files were found.

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

**~**

-- No entries --

**[email protected]** **~ $**

**[email protected]** **~ $** sudo systemctl status letsencrypt-init.service

● letsencrypt-init.service - LetsEncrypt Init Service

Loaded: loaded (/etc/systemd/system/letsencrypt-init.service; disabled; vendor preset: disabled)

Active: inactive (dead) since Tue 2020-06-23 22:02:42 UTC; 15h ago

Condition: start **condition failed** at Wed 2020-06-24 13:07:12 UTC; 22s ago

└─ ConditionPathExists=!/tmp/letsencrypt.init was not met

 Main PID: 17137 (code=exited, status=0/SUCCESS)

**[email protected]** **~ $**