signature=41002d8186eebe290c5b32b454d59514,Registry doesn't accept valid JWT auth token "error while...

Problem description

NOTE the authentication with the docker daemon and client works in my setup only if I want to do it manually, there seem to be an issue which I describe below.

To access my secured self-hosted registry (v2) with curl I have to follow a few steps:

I query my registry's /v2/_catalog route and get an HTTP/1.1 401 Unauthorized

I request a token from my auth server with the service and scope as returned in the response on 1.

I pass the token as Authorization: Bearer MYTOKEN to my docker registry when requesting the /v2/_catalog route.

Everything is fine except that the registry won't accept my token due to this error:

time="2015-10-13T09:35:49Z" level=error msg="error while unmarshalling raw token: unable to decode signature: illegal base64url string"

time="2015-10-13T09:35:49Z" level=warning msg="error authorizing context: malformed token" http.request.host="127.0.0.1:5000" http.request.id=e41329cd-760b-4e4a-89ff-5f726b77fd0d http.request.method=GET http.request.remoteaddr="172.17.42.1:55231" http.request.uri="/v2/_catalog" http.request.useragent="curl/7.35.0" instance.id=75c575c0-6b14-4c90-8cdf-c1910f42ad47 version=v2.1.1

172.17.42.1 - - [13/Oct/2015:09:35:49 +0000] "GET /v2/_catalog HTTP/1.1" 401 161 "" "curl/7.35.0"

I've double checked that my token is valid using the JWT Debugger and pasting the token as well as the certifiate and the private key (no real secret since it is a demo cert/key which can be found here):

Background

I have secured my registry using a token server. My exact setup can be found here.

Expected results

The registry accepts my token.

Actual result

Registry complains about malformed token.

Detailed steps

1. Request token

$ curl -H "Authorization: Basic MYBASE64ENCODEDSECRET" -vk "https://127.0.0.1:5001/auth?service=my.docker.registry&scope=registry:catalog:*"

* Hostname was NOT found in DNS cache

* Trying 127.0.0.1...

* Connected to 127.0.0.1 (127.0.0.1) port 5001 (#0)

* successfully set certificate verify locations:

* CAfile: none

CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Server key exchange (12):

* SSLv3, TLS handshake, Server finished (14):

* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSL connection using ECDHE-RSA-AES128-GCM-SHA256

* Server certificate:

* subject: C=AU; ST=Example State; L=Example city; O=Example company; OU=Example unit; CN=example.com; emailAddress=admin@example.com

* start date: 2015-09-16 13:10:20 GMT

* expire date: 2016-09-15 13:10:20 GMT

* issuer: C=AU; ST=Example State; L=Example city; O=Example company; OU=Example unit; CN=example.com; emailAddress=admin@example.com

* SSL certificate verify ok.

> GET /auth?service=my.docker.registry&scope=registry:catalog:* HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 127.0.0.1:5001

> Accept: */*

> Authorization: Basic MYBASE64ENCODEDSECRET

>

< HTTP/1.1 200 OK

< Content-Type: application/json

< Date: Tue, 13 Oct 2015 09:33:51 GMT

< Content-Length: 1108

<

{"token":"eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkdWNU06M0FGVDpaUVRYOlhHQUI6TFlZWDo1S1I1Olg2RUw6RTZTTzo3Q0E0OkJBWUU6RUdGTjo2UEtYIn0.eyJpc3MiOiJBY21lIGF1dGggc2VydmVyIiwic3ViIjoia29ucmFka2wiLCJhdWQiOiJteS5kb2NrZXIucmVnaXN0cnkiLCJleHAiOjE0NDQ3Mjk3MzAsIm5iZiI6MTQ0NDcyODgyOSwiaWF0IjoxNDQ0NzI4ODMwLCJqdGkiOiI3ODY2MDE5Mjk2MDYwMzY0MjYxIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVnaXN0cnkiLCJuYW1lIjoiY2F0YWxvZyIsImFjdGlvbnMiOlsiKiJdfV19.LSxAbsi1Okh3cjtEwEu1wl0aRjlVj7y-e7Ur2srvMyGKr--u1iwd5NANg_ZZiORjGt5MzSlEI9YDWrAoo2iZLCz80OWO7E1p32X7__4tsOEgOQFEBu5snrJJH64HDMcAFXKhXODP2h1G1Nm7eEz5K66MDn49nNe32fYBo8VUBXuHWuiVSox8YRUiPe4mVTt6nQ2iLbIdttsloMGSRHPp216ntgr9JGlrVv5dHt1Lkb8jCoIKToXzT5ctzqQkk-N1-yUoQPVhD5M624EUBb_II3iY6lAQ2A4CUILdrbx4nL8vLQOadt-N24wnbcwDCivf2hxmJTGEm5UYUeAICAwgj-bdPNw0F4lWGkaQiNYUspp_VhItiYRxXx3FW3MnOMM21BZoYPwrRA9JXlWYyNF097WGNjAI8J5284hf6QvQMK90ivNeNY7U-3CBLnuER4-6KEl5GlnQ5VVF-LfGkFS3i-cWa_5OizbaPZ9-onscOvrgsk7kboHu84jBUqh0Lo5_0a6Q1O0K0fjmn_90L231yvRJ8cz4bgxdUPdbTmmrkCi2UjMtQlWzKc7MqHDZWr7Rwv7jliG_dwZKZfu11YkMn0an8* Connection #0 to host 127.0.0.1 left intact

JgSmMfDt_SIHdeu8pzEXLvUYmpu0gIa6XwfZfooNTvijYXJ8Wsy5xAltE8EEXxmd1zdCcjhMlC3drEZ8EA"}

2. Using the token to request catalog from docker registry

curl -vk -H "Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkdWNU06M0FGVDpaUVRYOlhHQUI6TFlZWDo1S1I1Olg2RUw6RTZTTzo3Q0E0OkJBWUU6RUdGTjo2UEtYIn0.eyJpc3MiOiJBY21lIGF1dGggc2VydmVyIiwic3ViIjoia29ucmFka2wiLCJhdWQiOiJteS5kb2NrZXIucmVnaXN0cnkiLCJleHAiOjE0NDQ3Mjk3MzAsIm5iZiI6MTQ0NDcyODgyOSwiaWF0IjoxNDQ0NzI4ODMwLCJqdGkiOiI3ODY2MDE5Mjk2MDYwMzY0MjYxIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVnaXN0cnkiLCJuYW1lIjoiY2F0YWxvZyIsImFjdGlvbnMiOlsiKiJdfV19.LSxAbsi1Okh3cjtEwEu1wl0aRjlVj7y-e7Ur2srvMyGKr--u1iwd5NANg_ZZiORjGt5MzSlEI9YDWrAoo2iZLCz80OWO7E1p32X7__4tsOEgOQFEBu5snrJJH64HDMcAFXKhXODP2h1G1Nm7eEz5K66MDn49nNe32fYBo8VUBXuHWuiVSox8YRUiPe4mVTt6nQ2iLbIdttsloMGSRHPp216ntgr9JGlrVv5dHt1Lkb8jCoIKToXzT5ctzqQkk-N1-yUoQPVhD5M624EUBb_II3iY6lAQ2A4CUILdrbx4nL8vLQOadt-N24wnbcwDCivf2hxmJTGEm5UYUeAICAwgj-bdPNw0F4lWGkaQiNYUspp_VhItiYRxXx3FW3MnOMM21BZoYPwrRA9JXlWYyNF097WGNjAI8J5284hf6QvQMK90ivNeNY7U-3CBLnuER4-6KEl5GlnQ5VVF-LfGkFS3i-cWa_5OizbaPZ9-onscOvrgsk7kboHu84jBUqh0Lo5_0a6Q1O0K0fjmn_90L231yvRJ8cz4bgxdUPdbTmmrkCi2UjMtQlWzKc7MqHDZWr7Rwv7jliG_dwZKZfu11YkMn0an8" https://127.0.0.1:5000/v2/_catalog

* Hostname was NOT found in DNS cache

* Trying 127.0.0.1...

* Connected to 127.0.0.1 (127.0.0.1) port 5000 (#0)

* successfully set certificate verify locations:

* CAfile: none

CApath: /etc/ssl/certs

* SSLv3, TLS handshake, Client hello (1):

* SSLv3, TLS handshake, Server hello (2):

* SSLv3, TLS handshake, CERT (11):

* SSLv3, TLS handshake, Server key exchange (12):

* SSLv3, TLS handshake, Server finished (14):

* SSLv3, TLS handshake, Client key exchange (16):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSLv3, TLS change cipher, Client hello (1):

* SSLv3, TLS handshake, Finished (20):

* SSL connection using ECDHE-RSA-AES128-GCM-SHA256

* Server certificate:

* subject: C=AU; ST=Example State; L=Example city; O=Example company; OU=Example unit; CN=example.com; emailAddress=admin@example.com

* start date: 2015-09-16 13:10:20 GMT

* expire date: 2016-09-15 13:10:20 GMT

* issuer: C=AU; ST=Example State; L=Example city; O=Example company; OU=Example unit; CN=example.com; emailAddress=admin@example.com

* SSL certificate verify ok.

> GET /v2/_catalog HTTP/1.1

> User-Agent: curl/7.35.0

> Host: 127.0.0.1:5000

> Accept: */*

> Authorization: Bearer eyJ0eXAiOiJKV1QiLCJhbGciOiJSUzI1NiIsImtpZCI6IkdWNU06M0FGVDpaUVRYOlhHQUI6TFlZWDo1S1I1Olg2RUw6RTZTTzo3Q0E0OkJBWUU6RUdGTjo2UEtYIn0.eyJpc3MiOiJBY21lIGF1dGggc2VydmVyIiwic3ViIjoia29ucmFka2wiLCJhdWQiOiJteS5kb2NrZXIucmVnaXN0cnkiLCJleHAiOjE0NDQ3Mjk3MzAsIm5iZiI6MTQ0NDcyODgyOSwiaWF0IjoxNDQ0NzI4ODMwLCJqdGkiOiI3ODY2MDE5Mjk2MDYwMzY0MjYxIiwiYWNjZXNzIjpbeyJ0eXBlIjoicmVnaXN0cnkiLCJuYW1lIjoiY2F0YWxvZyIsImFjdGlvbnMiOlsiKiJdfV19.LSxAbsi1Okh3cjtEwEu1wl0aRjlVj7y-e7Ur2srvMyGKr--u1iwd5NANg_ZZiORjGt5MzSlEI9YDWrAoo2iZLCz80OWO7E1p32X7__4tsOEgOQFEBu5snrJJH64HDMcAFXKhXODP2h1G1Nm7eEz5K66MDn49nNe32fYBo8VUBXuHWuiVSox8YRUiPe4mVTt6nQ2iLbIdttsloMGSRHPp216ntgr9JGlrVv5dHt1Lkb8jCoIKToXzT5ctzqQkk-N1-yUoQPVhD5M624EUBb_II3iY6lAQ2A4CUILdrbx4nL8vLQOadt-N24wnbcwDCivf2hxmJTGEm5UYUeAICAwgj-bdPNw0F4lWGkaQiNYUspp_VhItiYRxXx3FW3MnOMM21BZoYPwrRA9JXlWYyNF097WGNjAI8J5284hf6QvQMK90ivNeNY7U-3CBLnuER4-6KEl5GlnQ5VVF-LfGkFS3i-cWa_5OizbaPZ9-onscOvrgsk7kboHu84jBUqh0Lo5_0a6Q1O0K0fjmn_90L231yvRJ8cz4bgxdUPdbTmmrkCi2UjMtQlWzKc7MqHDZWr7Rwv7jliG_dwZKZfu11YkMn0an8

>

< HTTP/1.1 401 Unauthorized

< Content-Type: application/json; charset=utf-8

< Docker-Distribution-Api-Version: registry/2.0

< Www-Authenticate: Bearer realm="https://0.0.0.0:5001/auth",service="my.docker.registry",scope="registry:catalog:*",error="invalid_token"

< Date: Tue, 13 Oct 2015 09:35:49 GMT

< Content-Length: 161

<

{"errors":[{"code":"UNAUTHORIZED","message":"access to the requested resource is not authorized","detail":[{"Type":"registry","Name":"catalog","Action":"*"}]}]}

* Connection #0 to host 127.0.0.1 left intact

Error reported by the registry

time="2015-10-13T09:35:49Z" level=error msg="error while unmarshalling raw token: unable to decode signature: illegal base64url string"

time="2015-10-13T09:35:49Z" level=warning msg="error authorizing context: malformed token" http.request.host="127.0.0.1:5000" http.request.id=e41329cd-760b-4e4a-89ff-5f726b77fd0d http.request.method=GET http.request.remoteaddr="172.17.42.1:55231" http.request.uri="/v2/_catalog" http.request.useragent="curl/7.35.0" instance.id=75c575c0-6b14-4c90-8cdf-c1910f42ad47 version=v2.1.1

172.17.42.1 - - [13/Oct/2015:09:35:49 +0000] "GET /v2/_catalog HTTP/1.1" 401 161 "" "curl/7.35.0"

Docker info

$ sudo docker info

Containers: 27

Images: 468

Storage Driver: aufs

Root Dir: /var/lib/docker/aufs

Backing Filesystem: extfs

Dirs: 522

Dirperm1 Supported: true

Execution Driver: native-0.2

Logging Driver: json-file

Kernel Version: 3.19.0-30-generic

Operating System: Ubuntu 14.04.3 LTS

CPUs: 8

Total Memory: 15.63 GiB

Name: kkleinework

ID: A5PC:FL5L:UJM6:4P7S:KXTW:B3PT:74OK:M3OL:DOJD:DDGH:DQK3:6FLY

Debug mode (server): true

File Descriptors: 42

Goroutines: 58

System Time: 2015-10-13T12:05:49.419286601+02:00

EventsListeners: 0

Init SHA1: 1f4a3c648015cae3b3d76c5ba2980d8c1f88f388

Init Path: /usr/lib/docker/dockerinit

Docker Root Dir: /var/lib/docker

WARNING: No swap limit support