Server is not responding on Mac M1

[PandaGab]

Hello,
When I try to log in after a fresh installation, I get the following error:

Thank you for you help,
Gabriel

[joshmoore]

What does:

docker-compose ps

show you? If one of the hosts has shutdown, we'll likely need to look into the log files. (One possibility is that there isn't enough memory)

[PandaGab]

Output from docker-compose ps:

NAME                                 COMMAND                  SERVICE             STATUS              PORTS
docker-example-omero_database_1      "docker-entrypoint.s…"   database            running             5432/tcp
docker-example-omero_omeroserver_1   "/usr/local/bin/entr…"   omeroserver         running             0.0.0.0:4063-4064->4063-4064/tcp
docker-example-omero_omeroweb_1      "/usr/local/bin/entr…"   omeroweb            running             0.0.0.0:4080->4080/tcp

Here is all the logs:

omero-server logs

Running /startup/50-config.py 
OpenSSL 1.0.2k-fips  26 Jan 2017
certificates created: /OMERO/certs/server.pem /OMERO/certs/server.p12
Running /startup/60-database.sh 
postgres connection established
Initialising database
5.11.2
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.DefaultDir=/OMERO/certs
INFO:omero_certificates.certificates:Setting omero.certificates.commonname=localhost
INFO:omero_certificates.certificates:Setting omero.certificates.owner=/L=OMERO/O=OMERO.server
INFO:omero_certificates.certificates:Setting omero.certificates.key=server.key
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.CertFile=server.p12
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.CAs=server.pem
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.Password=secret
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.Ciphers=HIGH
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.ProtocolVersionMax=TLS1_2
INFO:omero_certificates.certificates:Setting omero.glacier2.IceSSL.Protocols=TLS1_0,TLS1_1,TLS1_2
INFO:omero_certificates.certificates:Executing: openssl version
INFO:omero_certificates.certificates:Using existing key: /OMERO/certs/server.key
INFO:omero_certificates.certificates:Creating self-signed certificate: /OMERO/certs/server.pem
INFO:omero_certificates.certificates:Executing: openssl req -new -x509 -subj /L=OMERO/O=OMERO.server/CN=localhost -days 365 -key /OMERO/certs/server.key -out /OMERO/certs/server.pem -extensions v3_ca
INFO:omero_certificates.certificates:Creating PKCS12 bundle: /OMERO/certs/server.p12
INFO:omero_certificates.certificates:Executing: openssl pkcs12 -export -out /OMERO/certs/server.p12 -inkey /OMERO/certs/server.key -in /OMERO/certs/server.pem -name server -password pass:secret
2022-10-03 21:01:50,140 [omego.extern] INFO  Executing : /opt/omero/server/OMERO.server/bin/omero version
OMERO.py version:
OMERO.server version:
5.6.5-ice36-b233
2022-10-03 21:01:57,812 [omego.extern] INFO  Completed [7.665 s]
2022-10-03 21:01:57,818 [    omego.db] INFO  DbAdmin: DbAdmin OMERO.server ...
2022-10-03 21:01:57,819 [omego.extern] INFO  Running [current environment]: /opt/omero/server/OMERO.server/bin/omero config get --show-password
2022-10-03 21:01:57,820 [omego.extern] INFO  Executing : /opt/omero/server/OMERO.server/bin/omero config get --show-password
2022-10-03 21:02:04,420 [omego.extern] INFO  Completed [6.598 s]
2022-10-03 21:02:04,425 [omego.extern] INFO  Executing [custom environment]: psql -v ON_ERROR_STOP=on -d omero -h database -U omero -w -A -t --version
2022-10-03 21:02:04,465 [omego.extern] INFO  Completed [0.039 s]
2022-10-03 21:02:04,466 [    omego.db] INFO  psql version: psql (PostgreSQL) 11.16
2022-10-03 21:02:04,467 [omego.extern] INFO  Running [current environment]: /opt/omero/server/OMERO.server/bin/omero config get --show-password

2022-10-03 21:02:04,467 [omego.extern] INFO  Executing : /opt/omero/server/OMERO.server/bin/omero config get --show-password

2022-10-03 21:02:09,388 [omego.extern] INFO  Completed [4.921 s]

2022-10-03 21:02:09,390 [omego.extern] INFO  Executing [custom environment]: psql -v ON_ERROR_STOP=on -d omero -h database -U omero -w -A -t -c \conninfo

2022-10-03 21:02:09,439 [omego.extern] INFO  Completed [0.049 s]

2022-10-03 21:02:09,442 [    omego.db] INFO  Creating SQL: omero-20221003-210209-440521.sql

2022-10-03 21:02:09,443 [omego.extern] INFO  Running [current environment]: /opt/omero/server/OMERO.server/bin/omero db script -f omero-20221003-210209-440521.sql   omero

2022-10-03 21:02:09,443 [omego.extern] INFO  Executing : /opt/omero/server/OMERO.server/bin/omero db script -f omero-20221003-210209-440521.sql   omero

2022-10-03 21:02:22,751 [omego.extern] INFO  Completed [13.306 s]

2022-10-03 21:02:22,755 [    omego.db] INFO  Creating database using omero-20221003-210209-440521.sql

2022-10-03 21:02:22,756 [omego.extern] INFO  Running [current environment]: /opt/omero/server/OMERO.server/bin/omero config get --show-password

2022-10-03 21:02:22,756 [omego.extern] INFO  Executing : /opt/omero/server/OMERO.server/bin/omero config get --show-password

2022-10-03 21:02:27,247 [omego.extern] INFO  Completed [4.490 s]

2022-10-03 21:02:27,249 [omego.extern] INFO  Executing [custom environment]: psql -v ON_ERROR_STOP=on -d omero -h database -U omero -w -A -t -f omero-20221003-210209-440521.sql

2022-10-03 21:02:28,976 [omego.extern] INFO  Completed [1.727 s]

2022-10-03 21:02:28,978 [    omego.db] WARNI stderr: b'psql:omero-20221003-210209-440521.sql:2842: NOTICE:  identifier "fkcontraststretchingcontext_codomainmapcontext_id_codomainmapcontext" will be truncated to "fkcontraststretchingcontext_codomainmapcontext_id_codomainmapco"\npsql:omero-20221003-210209-440521.sql:4712: NOTICE:  identifier "fklogicalchannel_photometricinterpretation_photometricinterpretation" will be truncated to "fklogicalchannel_photometricinterpretation_photometricinterpret"\npsql:omero-20221003-210209-440521.sql:5697: NOTICE:  identifier "fkreverseintensitycontext_codomainmapcontext_id_codomainmapcontext" will be truncated to "fkreverseintensitycontext_codomainmapcontext_id_codomainmapcont"\n'

Running /startup/99-run.sh

Starting OMERO.server

omero-database logs

The files belonging to this database system will be owned by user "postgres".
This user must also own the server process.
The database cluster will be initialized with locale "en_US.utf8".

The default database encoding has accordingly been set to "UTF8".

The default text search configuration will be set to "english".
Data page checksums are disabled.
fixing permissions on existing directory /var/lib/postgresql/data ... ok

creating subdirectories ... ok

selecting default max_connections ... 100

selecting default shared_buffers ... 128MB

selecting default timezone ... Etc/UTC

selecting dynamic shared memory implementation ... posix

creating configuration files ... ok

running bootstrap script ... ok

performing post-bootstrap initialization ... ok

syncing data to disk ... ok
Success. You can now start the database server using:
pg_ctl -D /var/lib/postgresql/data -l logfile start

waiting for server to start....2022-10-03 21:01:23.136 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2022-10-03 21:01:23.145 UTC [49] LOG:  database system was shut down at 2022-10-03 21:01:22 UTC

2022-10-03 21:01:23.150 UTC [48] LOG:  database system is ready to accept connections

done

server started

CREATE DATABASE
/usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
2022-10-03 21:01:23.579 UTC [48] LOG:  received fast shutdown request

waiting for server to shut down....2022-10-03 21:01:23.580 UTC [48] LOG:  aborting any active transactions

2022-10-03 21:01:23.583 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1

2022-10-03 21:01:23.583 UTC [50] LOG:  shutting down

2022-10-03 21:01:23.595 UTC [48] LOG:  database system is shut down

done

server stopped
PostgreSQL init process complete; ready for start up.
WARNING: enabling "trust" authentication for local connections

You can change this by editing pg_hba.conf or using the option -A, or

--auth-local and --auth-host, the next time you run initdb.

2022-10-03 21:01:23.687 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432

2022-10-03 21:01:23.687 UTC [1] LOG:  listening on IPv6 address "::", port 5432

2022-10-03 21:01:23.689 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"

2022-10-03 21:01:23.701 UTC [76] LOG:  database system was shut down at 2022-10-03 21:01:23 UTC

2022-10-03 21:01:23.705 UTC [1] LOG:  database system is ready to accept connections

2022-10-03 21:01:48.522 UTC [84] ERROR:  relation "dbpatch" does not exist at character 15

2022-10-03 21:01:48.522 UTC [84] STATEMENT:  select * from dbpatch

omero-web logs

Running /startup/50-config.py 
Running /startup/60-default-web-config.sh 
Running /startup/98-cleanprevious.sh 
Running /startup/99-run.sh 
Starting OMERO.web
652 static files copied to '/opt/omero/web/OMERO.web/var/static', 2 post-processed.

Clearing expired sessions. This may take some time... [OK]

[2022-10-03 21:01:55 +0000] [84] [INFO] Starting gunicorn 20.1.0

[2022-10-03 21:01:55 +0000] [84] [INFO] Listening at: http://0.0.0.0:4080 (84)

[2022-10-03 21:01:55 +0000] [84] [INFO] Using worker: sync

[2022-10-03 21:01:55 +0000] [93] [INFO] Booting worker with pid: 93

[2022-10-03 21:01:55 +0000] [95] [INFO] Booting worker with pid: 95

[2022-10-03 21:01:55 +0000] [97] [INFO] Booting worker with pid: 97

[2022-10-03 21:01:56 +0000] [99] [INFO] Booting worker with pid: 99

[2022-10-03 21:01:56 +0000] [101] [INFO] Booting worker with pid: 101

[2022-10-03 21:10:32 +0000] [84] [CRITICAL] WORKER TIMEOUT (pid:97)

[2022-10-03 22:10:32 +0100] [97] [INFO] Worker exiting (pid: 97)

[2022-10-03 21:10:32 +0000] [138] [INFO] Booting worker with pid: 138

[joshmoore]

Hi @PandaGab,

thanks for the output. Those look fine. It would help to also take a look at the internal Blitz-0.log. Could you share the following with us?

docker-compose exec omeroserver cat /opt/omero/server/OMERO.server/var/log/Blitz-0.log

[PandaGab]

Hey !

Interestingly, there are no logs !

sh-4.2$ pwd
/opt/omero/server/OMERO.server/var
sh-4.2$ ls -la
total 12
drwxr-xr-x 2 omero-server omero-server 4096 Jun 29 08:56 .
drwxr-xr-x 1 omero-server omero-server 4096 Jun 29 08:56 ..

[joshmoore]

Well that's odd! Can you check how much available memory there is in the omeroserver container?

[PandaGab]

Sure !

sh-4.2$ whoami
omero-server
sh-4.2$ df -h /
Filesystem      Size  Used Avail Use% Mounted on
overlay         252G  7.5G  232G   4% /

[joshmoore]

Thanks. And free -m?

[PandaGab]

sh-4.2$ free -m
              total        used        free      shared  buff/cache   available
Mem:           7951        1308        5635         352        1007        6120
Swap:          4095           0        4095

[joshmoore]

Hmm.... that's a bit on the small side but I would think it would at least produce an error log. For reference:

docker-compose exec omeroserver /opt/omero/server/OMERO.server/bin/omero admin jvmcfg
JVM Settings:
============
blitz=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
indexer=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
pixeldata=-Xmx7200m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions
repository=-Xmx4800m -XX:MaxPermSize=1g -XX:+IgnoreUnrecognizedVMOptions

[PandaGab]

Thanks for the logs.

FYI : I tried on my old Mac (without the M1 chip) and got a working installation !

[joshmoore]

FYI : I tried an my old Mac (without the M1 chip) and got a working installation !

Glad to hear it, but really unfortunate that M1 is causing issues. If you (or anyone else) happens to find an error message or a log of some form, we'd love to see it.

[Helfrid]

Has there been any progress on this issue? When I run the docker-compose pull command I get: no matching manifest for linux/arm64/v8 in the manifest list entries. As with PandaGab I got it to work without problems on an intel Mac.

[joshmoore]

Hi @Helfrid. The "no matching manifest" issue has to do with there not being an arm build on docker hub. If you trying building the image locally, that should go away but then I assume you will run into the same problem as @PandaGab.

Looking at this with @will-moore, I assume what we are running into is that there isn't an upstream build of the Ice packages. My assumption from looking briefly is that the dependencies look like this:

• The docker build uses an ansible playbook.
• The ansible playbook uses ansible-role-ice.
• ansible-role-ice makes use of the redhat.yml tasks.
• The redhat.yml tasks use the Ice repository but there's no arm build.

The solution, I believe, is to make a repository like zeroc-ice-ubuntu1804 which builds zeroc on arm and (e.g., "zeroc-ice-centos7-arm") and provides a downloadable package.

If someone would like to test this theory (I don't have an m1 to help out 😬), then try a local build as suggested by @khaledk2 who tested on a Raspberry Pi (also arm):

Download ice 3.6.5 using the following command to your home folder:

git clone -b v3.6.5 https://github.com/zeroc-ice/ice.git
cd ice/cpp
make

See further instructions below.

Related issues:

• systemd problem ansible-example-omero-onenode#17
• This example on Mac M1 #12 (comment)

[khaledk2]

Sorry, my fault; there is no need for this line to build Ice locally:

tar xvfz Ice-3.6.5.tar.gz

Also, after the building you should update the path and library path to be able to use the build. You can do that by editing the .bash_profile on the Mac machine and adding these two lines by the end of the file:

export PATH=~/ice/cpp/bin:$PATH
export LD_LIBRARY_PATH=~/ice/cpp/lib:$LD_LIBRARY_PATH

You should run this command then everything should be ready.

source .bash_profile

[manics]

I've got some Ice packages compiled for Ubuntu arm64 in
https://github.com/manics/omero-ice-docker
https://forum.image.sc/t/omero-server-and-web-installation-on-linux-arm64/75628/3

The binaries are inside the container image because the idea was to use Docker's multi-arch support to compile Ice as part of a cross-architecture build, but (perhaps inevitably) it hit the 6 hour time-limit so I spun up a temporary arm64 GitHub runner on Oracle cloud instead. Given this cross-build idea didn't work you're better off following the standard zeroc-ice-ubuntuNNNN workflow linked above.

If you want a fully automated CI build you can sign up for a free Oracle Cloud account which includes "always* free" arm64 compute
https://www.oracle.com/uk/cloud/free/
and spinning up a runner (should be done in a private mirror of the repo for security)
https://github.com/manics/omero-ice-docker/blob/e2be1ddedf623286cbaa013d3569dc035f9ef0fd/.github/workflows/main.yml#L2-L4

(* interpretation of "always" is an exercise for the reader)