MQTT TLS on Google cloud - invalid JWT generated

I'm trying to connect to our thing on Google Cloud IoT Core, but I get an error when trying to connect using  

status = nxd_mqtt_client_secure_connect(&g_mqtt_client0, &broker_address, NXD_MQTT_TLS_PORT,
NX_MQTT_tls_setup, 0, NX_TRUE, NX_WAIT_FOREVER);

I figure out the cause is in nx_secure_tls_session_start.c, function _nx_secure_tls_handshake_process that Fails with 56 (0x38) error (NX_NOT_CONNECTED 0x38)

Then looking at the JWT that is generated, I've checked on jwt.io (a website) and gets me invalid JWT.

Note that I have RootCA certificate, device certificate and device private key (in pem format) and they work just fine with Python (just checking certificates and key are correct, and server address/ TLS is working), in fact the JWT generated in python gets me "genuine" under the same jwt.io website validator.

I dont' get any errors while loading certificates and key on netx secure, so I'm guessing is the way I pad or calculate the JWT is not correct, I'm using the same base64 and JWT create functions that you use in the MQTT_tls google cloud example project.

Root certificate and device certificate are converted first to .der (using openssl) and then to binary array using the hexy.exe tool.

Private key is converted first to pkcs1.pem, then to .der and finally to binary array.

All the CA certificate, device certificate and private key are loaded into some header files and included in my project.

Some code, after I establish an IP layer through Wifi, following:

//> SNTP gets time, I save timestamp and call get_time() when I need it (to generate JWT) - no errors here

//> DNS to get IP address of Google Cloud endpoint - no errors here

//>Init of mqtt packet pool and client - all good

Then init and store of certificates, I get no errors in the following:

status = nx_secure_tls_session_create(&g_mqtt_client0.nxd_mqtt_tls_session,
(NX_SECURE_TLS_CRYPTO *) &nx_crypto_tls_ciphers_synergys7,
crypto_metadata,
sizeof(crypto_metadata));

status = nx_secure_tls_session_packet_buffer_set(&g_mqtt_client0.nxd_mqtt_tls_session,
tls_packet_buffer,
sizeof(tls_packet_buffer));

//---------------------

char *temp_str = "unused";

for(UINT i = 0; i < sizeof(temp_str); i++)
{
username[i] = temp_str[i];
}

//Password is the JWT generated using realtime timestamp (epoch) and IoT thing private key
//rsa_private_pkcs1, RSA_PRIVATE_LEN
jwt_expiration = JWT_EXPIRATION_SECS;

/* Generate JWT */
if (jwt_create(password, sizeof(password), &jwt_expiration,
rsa_private_pkcs1, RSA_PRIVATE_LEN, (CHAR*) &project_id))
{
HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"Unable to create JWT. Aborting.\r\n");
__BKPT(0);
}

HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"JWT token created successfully %s\n============", password);
//set login
status = nxd_mqtt_client_login_set(&g_mqtt_client0, username, strlen(username),
(char *)password, strlen((char *)password));
if(status){
HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"tnxd_mqtt_client_login_set error %d", status);
__BKPT(0);
}

//---------------------------------- after this, I get back the error described above.

///> In nx_secure_tls_session_start.c
///> function _nx_secure_tls_handshake_process Fails with 56 (0x38) error
///> (NX_NOT_CONNECTED 0x38)

status = nxd_mqtt_client_secure_connect(&g_mqtt_client0, &broker_address, NXD_MQTT_TLS_PORT,
NX_MQTT_tls_setup, 0, NX_TRUE, NX_WAIT_FOREVER);

if(status){
HSP_APP_LOG(HSP_LOG_LEVEL_ERROR,"nxd_mqtt_client_secure_connect error %d", status);
__BKPT(0);
}

__BKPT(0);

I include my project file, so maybe somebody can address me on how to properly generate the JWT, as the following is not getting me a proper JWT.

https://www.filemail.com/d/hacpvrjcfpuxwnd

Is the certs/private key conversion and handling correct?

Do I need to change padding/ crypto functions in the JWT header/payload/signature generation?

Unfortunately I can not find any more info on this topic, as normally a jwt library is used to generate hash and create the token, but I want to make use of the hardware secure engine.

Any info/hints very welcome. 

I can share certs/keys if you want to check, but basically If I get a valid JWT generated I think we are half way there.

Generated JWT:

eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJoNG5oNDQtMSIsImV4cCI6MTYzMDQyMTE3OCwiaWF0IjoxNjMwNDIwODc4fQ==.SbNIiu23b0w9udd8jlICsW/cQqsbMgKvzQlSmga8mseI3hVAwW001YQ9QmLPc7Y3z8bE8ndYJXE3lO4UA39M4D6G/KyrybRnzQkFfevRm4qw8NekOsENPEse25YGNpzydKcVHvomHMBm8r797ApH2Z0Xs3FD2e+EgnS3GkQ4bYVxscMmDFqd/m5IWJPz8mZiwuN3+LXK2gVOF798wObXY7L7q8EWycV2QbeDOQlEkVyG1SFTwudM0Y+Geppsli5uUrscEq5ipT/rgIaIFuqUcvykH9IZm/sLHXh54UYIkaAlHdzGyWTr+50cV4X+9ZWwTcfQdrRcECKIsSgeC/JyHw==

(note the == characters generated, I suspect shouldn't be there)

and so: [0000002235][new_thread0_entry.c:0395] - nxd_mqtt_client_secure_connect error 65541

I'm on S5D9, SSP1.7.8 e2Studio. NetX Duo.

Thanks a lot.

S

  • Hi S,

    Have you try modifying  the Base64urlEncoder() in Base64Decode.c to strip the padding, "="? Would it pass checking jwt.io?

    The hash/signature would definitely be different without '=', As the Header/payload is available, the receiver could have use it to verify the signature,   

  • Hi,

    Yes I've tried but still doesn't pass the jwt.io check. Is there a valid Base64url encoder library that actually works with Google IoT Core?

    Also, it has something to do with the pkcs1 padding in the jwt_create() ?

    Thanks

  • Hi Rusty,

    I change (in Base64Decode.c)

    base64digits[] = "ABC...789+/"; 

    to

    base64digits[] = "ABC...789-_";

    last 2, '+', '/' change to '-', '_' 

    and in Base64urlEncoder() 

    '=' change to '\0'  to omit the last 1 or 2 padding. (2 instances of this)

            *out++ = (inlen < 2) ? '\0'  /*'='*/   : base64digits[(in[1] << 2) & 0x3c];
            *out++ = '\0';   //'=';

    Now the jwt.io verifies the JWT submitted has correct signature. 

    No changes to pkcs1 padding.

    Can you check if you can reach google server after these changes?

  • Hi Yep,

    I made the changes you suggested, but still I can not generate a valid JWT, here some extract from my logs and the base64 library I'm using.

    Any suggestions on how to get it working much appreciated.

    ----LOG----

    [hsp_wble.c:0690] - PPP init SUCCESS
    [hsp_wble.c:0692] - Wifi station link UP and PPP mode active
    [hsp_wble.c:0707] - Initialised NX common
    [hsp_wble.c:0826] - WBLE module init_connection
    [hsp_wble.c:0859] - Initialised IP
    [hsp_wble.c:0885] - Initialised PPP
    [hsp_wble.c:0908] - Initialised DNS
    [new_thread0_entry.c:0260] - START TIME/DATE >> Sep 2, 2021 12:32:54.841 UTC <<
    [new_thread0_entry.c:0321] - mqtt.2030.ltsapis.goog is at 173.194.76.206:8883
    [new_thread0_entry.c:0344] - Secure TLS session created!
    [new_thread0_entry.c:0375] - JWT token created successfully eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJhdWQiOiJoNG5oNDQtMSIsImV4cCI6MTYzMDU4NjI3NCwiaWF0IjoxNjMwNTg1OTc0fQ.ce4xJaVwO0HV4ybdnC8-vKTCUrHRroiYY_9R8egWW_izKQPGlUOxv6Ku2-ZLsquh7EJFcARPkg4EkewKymbQDCrBdBYND8Kyd6_NPF_A7nqPRcTwwUTHLOa9xzZxAI_xJpgpicAhGlM7kJCHDeerwSTNu6HWsi1E2uCjwM8NxuyWnsut9Lg12EXcBb8FJ2bF_km8QO6BGqHnlCbQf3byQ58IJ6LIaIy8buCgtu7GHc1FSVbmUHk263QwuFD99yGrh1k-91DkIf5QHUZHnGyn-r7bERQkYev8ef9m_gKiFDUz6sNZ3R1ibW62S6ufTrJGfgs6bQHCkI_n3JxcpCunaw

    [new_thread0_entry.c:0384] - MQTT Client login set successful
    [new_thread0_entry.c:0397] - nxd_mqtt_client_secure_connect error 0x10005

    ---Base64Decode.c

    #include <stdio.h>
    #include <string.h>
    #include <stdint.h>
    #include <assert.h>
    #include "base64.h"

    void Base64urlEncoder(unsigned char *out, const unsigned char *in, int inlen);

    //Base64 char table - used internally for encoding
    static const unsigned char base64digits[] =
    "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789-_";

    unsigned int base64_int(unsigned int ch)
    {

    /* ASCII to base64_int
    65-90 Upper Case >> 0-25
    97-122 Lower Case >> 26-51
    48-57 Numbers >> 52-61
    43 Plus (+) >> 62
    47 Slash (/) >> 63
    61 Equal (=) >> 64~
    */
    if (ch==43)
    return 62;
    if (ch==47)
    return 63;
    if (ch==61)
    return 64;
    if ((ch>47) && (ch<58))
    return ch + 4;
    if ((ch>64) && (ch<91))
    return ch - 'A';
    if ((ch>96) && (ch<123))
    return (ch - 'a') + 26;
    return 0;
    }

    unsigned int base64_decode(const unsigned char* in, unsigned int in_len, unsigned int* out)
    {

    unsigned int i=0, j=0, k=0, s[4] = {0};

    for (i=0;i<in_len;i++)
    {
    s[j++]=base64_int(*(in+i));
    if (j==4)
    {
    out[k+0] = ((s[0]&255)<<2)+((s[1]&0x30)>>4);
    if (s[2]!=64)
    {
    out[k+1] = ((s[1]&0x0F)<<4)+((s[2]&0x3C)>>2);
    if ((s[3]!=64))
    {
    out[k+2] = ((s[2]&0x03)<<6)+(s[3]); k+=3;
    }
    else
    {
    k+=2;
    }
    }
    else
    {
    k+=1;
    }
    j=0;
    }
    }

    return k;
    }

    unsigned int base64d_size(unsigned int in_size)
    {

    return ((3*in_size)/4);
    }

    ///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
    //
    // Function : Base64urlEncoder
    // Purpose : Converts a given string into a base64 encoded buffer.
    // Last updated : 01/09/200515/05/2005
    // Notes :
    //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

    void Base64urlEncoder(unsigned char *out, const unsigned char *in, int inlen)
    /* [OUT] out A pointer to a char to hold the converted string
    [IN] in String to convert
    [IN] inlen Length of the string to be converted
    */
    {
    for (; inlen >= 3; inlen -= 3)
    {
    *out++ = base64digits[in[0] >> 2];
    *out++ = base64digits[((in[0] << 4) & 0x30) | (in[1] >> 4)];
    *out++ = base64digits[((in[1] << 2) & 0x3c) | (in[2] >> 6)];
    *out++ = base64digits[in[2] & 0x3f];
    in += 3;
    }

    if (inlen > 0)
    {
    unsigned char fragment;

    *out++ = base64digits[in[0] >> 2];
    fragment = (in[0] << 4) & 0x30;

    if (inlen > 1)
    fragment |= in[1] >> 4;

    *out++ = base64digits[fragment];
    *out++ = (inlen < 2) ? '\0' : base64digits[(in[1] << 2) & 0x3c];
    *out++ = '\0';
    }

    *out = '\0';
    }

     

  • Correction: 

    The JWT is actually now verified as correct, using jwt.io and the device / private key.

    At this point what could be the cause?

    Timestamp is calculated correctly? (gives me epochs that when converted are in UTC+0)

    I've tried JWT expiration to 1 hour, 2 hours, but seems not to make a difference.

    Again, all the certificate work with a python mqtt project, and I can connect securely to the same broker "mqtt.2030.ltsapis.goog", on port 8883.

    Is there a way to see why this is failing, other that the generic error 0x10005 from nxd_mqtt_client_secure_connect()?

    Debugging to lower level I get that NX is not connected, even though I have an IP layer opened and working.

    ///> In nx_secure_tls_session_start.c
    ///> function _nx_secure_tls_handshake_process Fails with 56 (0x38) error
    ///> (NX_NOT_CONNECTED 0x38)

    Thanks

    my configuration: all IP instances are shared, TLS common is shared between MQTT and TLS Session:

  • Hi,

    Could you get a wireshark logs? Hope to understand why server terminates this tcp connection.

    Just to double check, is the size of your certificate correct?  You mention you convert this certificate into byte array. Could you do a memory dump of this and check if it's correct?

  • Hi Yep, 

    Yes I've done a capture of my wifi network from my PC while application running, the DHCP address assigned to my uBlox NINA is 192.168.1.12, something is going on but I'm not sure at what to look at.

    Following are my google certificate in PEM format (that I use from Python with no problems) and the corresponding byte array used in my FW. Size seems the same. 

    CA_cert.zip

    Should I use a different certificate or check if is loaded correctly into netx secure?

    Can I send the capture file in private to you? 

    Also, would it be possible to create a TLS session project (client) and use the same certificate just to see if it's loaded and checked correctly? I tried also an openssl s_client -connect with this certificate and broker endpoint and status returns 0 from openssl (verified, OK).

     Thanks,

    S

    edit: Looking here:AE-CLOUD2. Google cloud. Could not create tls session. - Synergy - Forum - Renesas Synergy Platform - RenesasRulz

    I have confirmed that I can connect to Azure and Amazon using the 2.1.0 SECT binary, but I cannot connect to Google. There has been a change in the Google certificate chain, that requires a different intermediate CA certificate to be used for Google. This would require a new SECT binary firmware.

    This answer maybe is a clue? Not sure how I load a certificate and an intermediate certificate, as a chain?

  • Hi Rusty,

    The CA_cert_der[] is an ECC cert.  Can you check if ECC is added to nx_crypto_tls_ciphers_synergys7. 

    Could you find/use RSA cert instead? You have install a root/intermediate cert as trusted cert.

    Other intermediate certs in the chain will be downloaded as needed to verify the server cert.

  • Hi,

    Yes all the suites are enabled, including ECC. 

    Do I need to specify that is an ECC cert in tls init?

    That certificate has been given by google, not sure If/how I can find an RSA version of it

  • Do you call nx_secure_tls_ecc_initialize() in the mqtt tls setup function?

    extern const USHORT     nx_crypto_ecc_supported_groups_synergys7[];
    extern const USHORT     nx_crypto_ecc_supported_groups_synergys7_size;
    extern const NX_CRYPTO_METHOD *     nx_crypto_ecc_curves_synergys7[];

     nx_secure_tls_ecc_initialize(p_tls_session,
                                              nx_crypto_ecc_supported_groups_synergys7,
                                              nx_crypto_ecc_supported_groups_synergys7_size,
                                              nx_crypto_ecc_curves_synergys7);

    in the mqtt tls setup function?

  • Hi Jeremy,

    No, I call x509 for both CA_cert and device cert. should I substitute the ca_cert one?

    Or add nx_secure_tls_ecc_initialize() at the beginning/end of the mqtt_tls_setup?

    Here's my mqtt tls setup function:

    static UINT NX_MQTT_tls_setup(NXD_MQTT_CLIENT *p_client, NX_SECURE_TLS_SESSION *p_tls_session,
    NX_SECURE_X509_CERT *p_cert, NX_SECURE_X509_CERT *p_trusted_cert)
    {
    UINT status = NX_SUCCESS;
    UINT index = 0;

    SSP_PARAMETER_NOT_USED (p_client);

    /* Need to allocate space for the certificate coming in from the remote host. */
    memset(&remote_cert_buffer, 0, sizeof(remote_cert_buffer));
    memset(&remote_certificate, 0, sizeof(remote_certificate));

    status = nx_secure_tls_remote_certificate_allocate(p_tls_session, &remote_certificate,
    remote_cert_buffer, sizeof(remote_cert_buffer));
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unabe to allocate memory for the certificate buffer %d", status);
    return status;
    }

    memset(remote_int_ca_cert_buffer, 0, sizeof(remote_int_ca_cert_buffer));
    memset(remote_intermediate_ca, 0, sizeof(remote_intermediate_ca));
    for (index = 0; index < ARRAY_SIZE(remote_intermediate_ca); index++)
    {
    status = nx_secure_tls_remote_certificate_allocate(p_tls_session, &(remote_intermediate_ca[index]),
    &(remote_int_ca_cert_buffer[index * CERTIFICATE_BUFFER_SIZE]),
    CERTIFICATE_BUFFER_SIZE);
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to allocate memory for interemediate CA certificate %d", status);
    return status;
    }
    }

    /* Add a CA Certificate to our trusted store for verifying incoming server certificates. */
    status = nx_secure_x509_certificate_initialize(p_trusted_cert,
    (UCHAR *) CA_cert_der, (USHORT) CA_CERT_DER_LEN,
    NX_NULL, 0, NULL, 0,
    NX_SECURE_X509_KEY_TYPE_RSA_PKCS1_DER);
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to initialize CA certificate %d", status);
    return status;
    }

    status = nx_secure_tls_trusted_certificate_add(p_tls_session, p_trusted_cert);
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to add CA certificate to trusted store, %d", status);
    return status;
    }

    if (dev_cert != NULL)
    {
    /* Add the local certificate for client authentication. */
    status = nx_secure_x509_certificate_initialize(p_cert, (UCHAR *) dev_cert, (USHORT) DEV_CERT_LEN,
    NX_NULL, 0, rsa_private_pkcs1, RSA_PRIVATE_LEN,
    NX_SECURE_X509_KEY_TYPE_RSA_PKCS1_DER);
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to initialize device certificate, %d",status);
    return status;
    }
    status = nx_secure_tls_local_certificate_add(p_tls_session, p_cert);
    if (NX_SUCCESS != status)
    {
    HSP_APP_LOG(HSP_LOG_LEVEL_DEBUG,"Unable to add device certificate to trusted store, %d",status);
    return status;
    }
    }

    /* Add a timestamp function for time checking and timestamps in the TLS handshake. */
    nx_secure_tls_session_time_function_set(p_tls_session, get_time);

    return status;
    }

    Thanks a lot

  • Do you call nx_secure_tls_ecc_initialize() anywhere? Try nx_secure_tls_ecc_initialize() adding after you call nx_secure_tls_session_create();

  • I've called secure_tls_ecc_initialize after session_create(), it returns 0 (I guess is NX_SUCCESS), but still I can not connect with the same 0x10005 error.

    I also tried to load the google certificate in RSA format (and changed broker to "mqtt.googleapis.com" as advised by IoT Core cloud.google.com/.../mqtt-bridge), but still I get the same error from mqtt connect. Also tried to use port 8883 and 443.

    www.filemail.com/.../cyvrwfffjslgbzp

  • Have you taken a Wireshark trace of the network activity to see what is going on in the communication between the Synergy board and the Google servers?

  • Yes, I did, 192.168.1.12 is my uBlox NINA Wifi DHCP address. Asus is our wifi router, external activity is captured from the internal wifi network but I think you can extract the big picture.

    Please find it here: www.filemail.com/.../mojofybpwdtrpuk

    I did also 2 additional captures with just my laptop (wireshark running) and uBlox connected to my WiFi hotspost.

    One using ECC certificate and the nx_secure_tls_ecc_initialize() (with the proper endpoint for ECC cert) and the RSA cert version.

    www.filemail.com/.../heenqzenkxqxpbx

    Interestingly, in these two last captures, I can't see the DNS request, even though I get it and resolve the IP of google cloud endpoint correctly from Synergy, also SNTP time query is not showing, and again I get the time correctly from the SNTP server.

    I think something is going on when resolving uBlox external IP, gateway IP, but again, I can perform SNTP queries, DNS queries, and HTTP Post successfully with the same stack.