Reading a SIM card phone book in Python

I already wrote a SIM phone book dumper program in 2004. This first version was in Perl and was presented in "SIM card phone book listing".

You can also find more advanced tools in articles with the SIM label, like cardpeek, monosim or PSSI.

I now present a version in Python using the PySCard wrapper

 

Source code

The source code is in 2 parts: usim.py and usim_read.py files.

usim.py is:

#!/usr/bin/env python3

from smartcard.System import readers
from smartcard.util import toBytes
from smartcard.CardConnectionObserver import ConsoleCardConnectionObserver

debug = True

def usim(reader_nb):
    # get all the available readers
    r = readers()
    print("Available readers:")
    for reader in r:
        print("-", reader)

    reader = r[reader_nb]
    print("Using:", reader)

    connection = reader.createConnection()
    if debug:
        observer = ConsoleCardConnectionObserver()
        connection.addObserver(observer)
    connection.connect()

    SELECT = "A0 A4 00 00 02 "
    GET_RESPONSE = "A0 C0 00 00 "

    # Select MF
    print("Select MF")
    data, sw1, sw2 = connection.transmit(toBytes(SELECT + "3F 00"))
    if sw1 != 0x9F:
        raise(Exception("Error"))

    # Select DF Telecom
    print("Select DF Telecom")
    data, sw1, sw2 = connection.transmit(toBytes(SELECT + "7F 10"))
    if sw1 != 0x9F:
        raise(Exception("Error"))

    # Select EF ADN
    print("Select EF ADN")
    data, sw1, sw2 = connection.transmit(toBytes(SELECT + "6F 3A"))
    if (sw1, sw2) != (0x9F, 0x0F):
        raise(Exception("Error"))

    # Get Response
    print("Get Response")
    data, sw1, sw2 = connection.transmit(toBytes(GET_RESPONSE) + [sw2])
    if (sw1, sw2) != (0x90, 0x00):
        raise(Exception("Error"))

    size = data[-1]

    pin = None
    if pin:
        print(pin)
        pin = list(map(ord, pin))
        padd(pin, 8)

        # Verify CHV
        VERIFY = "A0 20 00 01 08"
        cmd = toBytes(VERIFY) + pin
        data, sw1, sw2 = connection.transmit(cmd)
        if (sw1, sw2) != (0x90, 0x00):
            raise(Exception("Wrong PIN:" + pin))

    return size, connection


if __name__ == "__main__":
    usim(0)

usim_read.py is:

#!/usr/bin/env python3

from smartcard.util import toBytes, toASCIIString
import usim


def decode_record(record):
    """
    decode_record(toBytes("43 75 73 74 6F 6D 65 72 20 43 61 72 65 FF 06 A1 80 00 07 70 00 FF FF FF FF FF FF FF"))
    >> ['Customer Care', '0800700700']
    """
    X = len(record) - 14
    name = toASCIIString(record[0:X - 1]).replace("ÿ", "")
    # number of bytes for the phone number
    tel_size = record[X]
    phone = record[X + 2:X + tel_size + 1]

    decoded = ""
    for n in phone:
        hex = "%02X" % n
        high = hex[0]
        low = hex[1]
        decoded += low + high
    # if the number of digits is odd we suppress the padding
    if decoded[-1] == "F":
        decoded = decoded[:-1]
    phone = decoded

    return name, phone


def usim_read(reader_nb):
    # Select the EF ADN
    (size, connection) = usim.usim(reader_nb)

    for nbr in range(1, 250):
        #  Read record
        header = [0xA0, 0xB2]
        record_idx = nbr
        cmd = header + [record_idx, 0x04, size]
        data, sw1, sw2 = connection.transmit(cmd)
        if (sw1, sw2) != (0x90, 0x00):
            return

        name, phone = decode_record(data)
        if name != "":
            print(f"{record_idx}: Name: {name}, phone: {phone}")


if __name__ == "__main__":
    import sys
    if 2 == len(sys.argv):
        reader_nb = int(sys.argv[1])
    else:
        reader_nb = 0
    usim_read(reader_nb)


Comments

The debug is enabled. So you can see the communication between the application and the card.
You ned to change only one line to remove the APDU log if needed. 

The phone book record size is not fixed for all the SIM cards. The record size is returned in the last byte of the GET RESPONSE command after the SELECT EF (Elementary File) ADN (Abbreviated dialling numbers).

You can get more details about the EF ADN and the record coding in the document ETSI TS 131 102 V16.6.0 (2021-01) chapter "4.4.2.3 EF_ADN (Abbreviated dialling numbers)". It is important to note that ETSI (European Telecommunications Standards Institute) standards are public and free.

By default the first PC/SC reader is used. But you can select another reader by passing a number as argument to the program. We will use this feature later.

No PIN is defined and verified. I am using a sysmocom sysmoUSIM-SJS1 card. This card has the user PIN disabled by default.
You can enable PIN verification by defining a PIN in usim.py. The code is already present.

USIM (Universal Subscriber Identity Module) is a new application introduced for GSM version 3 (3G) to replace/improve the SIM (subscriber identity/identification) application. The code should work the same with a SIM card or a USIM card (but untested).

Output

$ ./usim_read.py 
Available readers:
- Gemalto PC Twin Reader
Using: Gemalto PC Twin Reader
connecting to Gemalto PC Twin Reader
Select MF
> A0 A4 00 00 02 3F 00
<  [] 9F 22
Select DF Telecom
> A0 A4 00 00 02 7F 10
<  [] 9F 22
Select EF ADN
> A0 A4 00 00 02 6F 3A
<  [] 9F 0F
Get Response
> A0 C0 00 00 0F
< 00 00 21 34 6F 3A 04 00 11 FF 22 01 02 01 22 90 00
> A0 B2 01 04 22
< 4C 61 75 72 65 20 46 72 61 6E E7 6F 69 73 65 20 59 76 6F 6E 06 A1 24 66 85 10 22 FF FF FF FF FF FF FF 90 00
1: Name: Laure Francoise Yvo, phone: 4266580122
> A0 B2 02 04 22
< 4C 75 63 69 65 6E 6E 65 20 48 65 6C 65 6E 65 20 4C 75 63 69 06 A1 76 45 65 28 77 FF FF FF FF FF FF FF 90 00
2: Name: Lucienne Helene Luc, phone: 6754568277
> A0 B2 03 04 22
< 55 72 62 61 69 6E 20 47 69 6C 6C 65 73 20 4A 75 73 74 65 FF 06 A1 80 38 74 16 54 FF FF FF FF FF FF FF 90 00
3: Name: Urbain Gilles Juste, phone: 0883476145
> A0 B2 04 04 22
< 4A 65 72 6F 6D 65 20 50 61 73 63 61 6C 20 46 65 72 6E 61 6E 06 A1 80 53 42 10 86 FF FF FF FF FF FF FF 90 00
4: Name: Jerome Pascal Ferna, phone: 0835240168
[...]

The output is truncated. I do not want to include all the 255 phone numbers. 

Note that the names and numbers are random. More on that later.


Conclusion

It is easy to dump the phone book from a SIM card.

The SIM phone book is very limited (no birthday, no email address). The real phone book is, in general, in the phone itself and synchronised using CardDav with a server like Nextcloud.

GitHub Sponsors: first year

In January 2020 I engaged in the GitHub Sponsors project. I got my first payment in May 2020. After a full year I think it is time for a public status. 

 

Results

My gains give a total of 296,82€ for 12 months. Of course I will have to pay some French taxes (employee and employer social security contributions URSSAF) of around 25% and also some income taxes.

Description
Revenue 296,82 €
Social Contributions
-74,20 €
Remaining
225,62 €

That is 18,55€/month.

 

Bills

That is not much (I can't live with that yet) but enough to pay some of the infrastructure used by my smart card projects: a Virtual Private Server to host the web sites and the domain name for https://muscle.apdu.fr/, https://pcsclite.apdu.fr/ and https://ccid.apdu.fr/.

Description
Virtual Private Server 100,66 €
Domain name .apdu.fr
8,39 €
Total
109,05 €

That is 9,09€/month.

 

Conclusion

Thank you to the generous sponsors.

New version of pcsc-lite: 1.9.1

I just released a new version of pcsc-lite 1.9.0.
pcsc-lite is a Free Software implementation of the PC/SC (or WinSCard) API for Unix systems.

 Changes:

1.9.1: Ludovic Rousseau
16 February 2021

  • Do not (possibly) lock a reader if allocating hCard fails
  • Fix a hang in SCardTransmit()
  • Do not report an error if the wrong interface is used by the driver
  • Update reader state when a card is removed during an exchange
  • readerfactory: Make sure a freed Reader Context is not accessed
  • PHSetProtocol(): supports T=0&1 cards on T=0 reader
  • hotplug-libusb:
    • support CCIDCLASSDRIVER
    • add interface name to reader name
    • remove obsolete libhal scheme
  • Some other minor improvements

New version of libccid: 1.4.34

I just released version 1.4.34 of libccid the Free Software CCID class smart card reader driver.

Changes

1.4.34 - 24 January 2021, Ludovic Rousseau
  • Add support of
    • ACS ACR1252IMP Reader
    • ACS CryptoMate EVO
    • Aktiv Rutoken SCR 3001 Reader
    • Avtor KP-375BLE
    • Avtor SC Reader KP382
    • BIT4ID mLector AIR DI V3
    • BIT4ID miniLector AIR NFC v3
    • Bit4id Digital-DNA Key (ProductID 0x2354)
    • Canokeys Canokey
    • DESKO GmbH IDenty chrom
    • DESKO GmbH PENTA Scanner
    • FT Biopass CCID
    • FT Biopass FIDO2
    • FT Biopass KB CCID
    • FT Biopass KB FIDO CCID
    • Feitian BLE CCID Dongle
    • Feitian R805
    • Feitian vR504 Contactless Reader
    • GoTrust Idem Key
    • Identiv uTrust 3720 Contactless Reader
    • Sunrex HP USB Business Slim Smartcard CCID Keyboard
    • sysmocom - s.f.m.c. GmbH sysmoOCTSIM
  • Fail if the requested protocol is not supported by reader
  • Disable USB suspend for the AlcorMicro AU9520 reader
  • Return "no smart card" if we get notified during a transmit
  • Minor improvements reported by Maksim Ivanov
  • Some other minor improvements

macOS Big Sur and smart card source code

Apple released the source code of the open source components they use in Big Sur (macOS 11.0, released in October 2020). The components are available at macOS X 11.0.1 Source.


SmartcardCCID

The SmartcardCCID component moved from version SmartcardCCID-55018.0.2 in Catalina 10.15.0 to SmartcardCCID-55021.40.1 in Big Sur 11.0.1.

Incomplete diff:

diff -ru SmartcardCCID-55018.0.2/SmartcardCCID.plist SmartcardCCID-55021.40.1/SmartcardCCID.plist
--- SmartcardCCID-55018.0.2/SmartcardCCID.plist	2019-08-21 00:16:22.000000000 +0200
+++ SmartcardCCID-55021.40.1/SmartcardCCID.plist	2020-04-28 20:53:09.000000000 +0200
@@ -6,13 +6,13 @@
 		<key>OpenSourceProject</key>
 		<string>ccid</string>
 		<key>OpenSourceVersion</key>
-		<string>1.4.31</string>
+		<string>1.4.32</string>
 		<key>OpenSourceWebsiteURL</key>
 		<string>https://ccid.apdu.fr</string>
 		<key>OpenSourceURL</key>
-		<string>https://ccid.apdu.fr/files/ccid-1.4.31.tar.bz2</string>
+		<string>https://ccid.apdu.fr/files/ccid-1.4.32.tar.bz2</string>
 		<key>OpenSourceImportDate</key>
-		<string>2019-08-20</string>
+		<string>2020-04-27</string>
 		<key>OpenSourceModifications</key>
 		<array>
 			<string>destDirFix.patch - makefile.in, customized destination directory</string>
diff -ru SmartcardCCID-55018.0.2/ccid/Makefile SmartcardCCID-55021.40.1/ccid/Makefile
--- SmartcardCCID-55018.0.2/ccid/Makefile	2019-08-21 00:16:21.000000000 +0200
+++ SmartcardCCID-55021.40.1/ccid/Makefile	2020-08-06 20:06:44.000000000 +0200
@@ -24,11 +24,12 @@
 	find $(DSTROOT)/ -name 'usb*.h' -exec rm \{\} \;
 	rm -r $(DSTROOT)/usr/include
 	rm -r $(DSTROOT)/usr/lib
-	install_name_tool -id $(CCIDDriversPath)$(CCIDdylib) $(DSTROOT)$(CCIDDriversPath)$(CCIDdylib) 
+	install_name_tool -id $(CCIDDriversPath)$(CCIDdylib) $(DSTROOT)$(CCIDDriversPath)$(CCIDdylib)
+	codesign -s - $(DSTROOT)$(CCIDDriversPath)$(CCIDdylib)
 
 # Automatic Extract & Patch
 AEP_Project    = ccid
-AEP_Version    = 1.4.31
+AEP_Version    = 1.4.32
 AEP_ProjVers   = $(AEP_Project)-$(AEP_Version)
 AEP_Filename   = $(AEP_ProjVers).tar.bz2
 AEP_ExtractDir = $(AEP_ProjVers)
[...]

As we already saw in macOS Big Sur and smart cards status the CCID driver was updated from version 1.4.31 to version 1.4.32. You can find the patches Apple applies to the CCID driver in the ccid/files/ directory. Nothing special to say.

In fact, after checking the different releases of Catalina 10.15.x in https://opensource.apple.com/ I found that the CCID driver was upgraded from 1.4.31 to 1.4.32 in Catalina itself from 10.15.5 to 10.15.6.

So Apple upgraded the CCID driver within the same major version version of macOS.
And they missed the opportunity to upgrade to 1.4.33 in Big Sur. Maybe it is planned for a future minor version upgrade of Big Sur?


libusb

SmartcardCCID includes the libusb component used by the CCID driver.

This libusb library is statically linked to the CCID driver and can't be used by another project.

The version is 1.0.9. This is a very old version of libusb that was released in April 2012. The current libusb version is 1.0.24 released in December 2020.

I guess Apple does not want to upgrade a component that works fine enough from them.


SecurityTokend

This component is the same as in Catalina. It is SecurityTokend-55113.

It is strange to still find a tokend related component. Tokend technology is deprecated since Mac OS X Lion in 2011 (Mac OS X Lion and tokend).

Tokend was disabled by default in Catalina but was still usable (macOS Catalina and smart cards status).

In Big Sur tokend are not usable at all.

This component SecurityTokend does not contain any tokend plugin. There were in the Tokend component, not SecurityTokend. This component generates two file: SecurityTokend.framework and libsecurity_tokend_client.a. I am not sure what they are used for.


Conclusion

Interesting parts of the smart card stack would be the CryptoTokenKit and WinSCard layers. But since Apple moved away from the Free Software project pcsc-lite in macOS Yosemite in 2014 (OS X Yosemite and smart cards status) these components are not open source.

MUSCLE mailing list statistics for 2020

As I did in 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018 and 2019 I propose some statistics of the MUSCLE mailing list usage.

Evolution

Year Total number of messages Progression
2009 603
2010 718 +19 %
2011 999 +39 %
2012 207 -79 %
2013 198 -4 %
2014 194 -2 %
2014 194 -2 %
2015 120 -38 %
2016 125 +4 %
2017 128 +2 %
2018 66 -51 %
2019 29 -56 %
2020 76 +162 %

Comments

A high increase in the number of messages. Maybe the mailing list is not dead after all.

Statistics from 3.1.2020 to 24.11.2020
for pcsclite-muscle@lists.infradead.org


People who have written most messages:

   Author   Msg   Percent 
1 ludovic.rousseau@gmail.com 36 47.37 %
2 Marc.Kewitz.ext@rohde-schwarz.com 4 5.26 %
3 axel.braun@gmx.de 4 5.26 %
4 trenta.sis@gmail.com 3 3.95 %
5 emaxx@google.com 3 3.95 %
6 suffsuccotash@gmail.com 3 3.95 %
7 stephan.guilloux@crisalid.com 3 3.95 %
8 zeroconf@zeroconf.ee 3 3.95 %
9 thomas@schlenkhoff.me 2 2.63 %
10 izh1979@gmail.com 2 2.63 %
11 sebastien@lorquet.fr 2 2.63 %
12 gregor.waltz@raritan.com 1 1.32 %
13 gsvelto@mozilla.com 1 1.32 %
14 Gabriele Svelto 1 1.32 %
15 tristan.degroof@multiversum.broedersvanliefde.be 1 1.32 %
16 t-pcsc@girst.at 1 1.32 %
17 mstjohns@comcast.net 1 1.32 %
18 mauro@faresoftware.it 1 1.32 %
19 Axel.braun@gmx.de 1 1.32 %
20 godfreyhkchung@gmail.com 1 1.32 %
21 jonathan.verner@nexusgroup.com 1 1.32 %
  other 1 1.32 %

Best authors, by total size of their messages (w/o quoting):

   Author   KBytes 
1 ludovic.rousseau@gmail.com 1032.8
2 stephan.guilloux@crisalid.com 38.4
3 zeroconf@zeroconf.ee 31.9
4 emaxx@google.com 31.7
5 Marc.Kewitz.ext@rohde-schwarz.com 30.6
6 sebastien@lorquet.fr 29.5
7 izh1979@gmail.com 23.3
8 suffsuccotash@gmail.com 17.2
9 gsvelto@mozilla.com 13.6
10 trenta.sis@gmail.com 12.4
11 mstjohns@comcast.net 11.3
12 axel.braun@gmx.de 10.8
13 godfreyhkchung@gmail.com 10.0
14 t-pcsc@girst.at 9.7
15 Gabriele Svelto 8.2
16 Axel.braun@gmx.de 4.1
17 thomas@schlenkhoff.me 2.9
18 tristan.degroof@multiversum.broedersvanliefde.be 2.7
19 jonathan.verner@nexusgroup.com 2.0
20 gregor.waltz@raritan.com 1.4
21 mauro@faresoftware.it 1.1

Best authors, by average size of their message (w/o quoting):

   Author   bytes 
1 ludovic.rousseau@gmail.com 29376
2 sebastien@lorquet.fr 15120
3 gsvelto@mozilla.com 13970
4 stephan.guilloux@crisalid.com 13096
5 izh1979@gmail.com 11923
6 mstjohns@comcast.net 11535
7 zeroconf@zeroconf.ee 10893
8 emaxx@google.com 10814
9 godfreyhkchung@gmail.com 10216
10 t-pcsc@girst.at 9922
11 Gabriele Svelto 8439
12 Marc.Kewitz.ext@rohde-schwarz.com 7844
13 suffsuccotash@gmail.com 5869
14 trenta.sis@gmail.com 4218
15 Axel.braun@gmx.de 4210
16 tristan.degroof@multiversum.broedersvanliefde.be 2798
17 axel.braun@gmx.de 2753
18 jonathan.verner@nexusgroup.com 2032
19 thomas@schlenkhoff.me 1490
20 gregor.waltz@raritan.com 1462
21 mauro@faresoftware.it 1143

Table showing the most successful subjects:

   Subject   Msg   Percent 
1 [Pcsclite-muscle] Workaround for missing suspend/resume 9 11.84 %
2 [Pcsclite-muscle] Different behaviour with Select MF on Omnikey 7 9.21 %
3 [Pcsclite-muscle] Alcor Micro AU9520 not recognized by libccid 1.4.31 7 9.21 %
4 [Pcsclite-muscle] Fwd: dead lock caused by SCardReleaseContext 4 5.26 %
5 [Pcsclite-muscle] memory consumption 3 3.95 %
6 [Pcsclite-muscle] pcsc-lite release 3 3.95 %
7 [Pcsclite-muscle] Exclusive access mode to the smart card reader 3 3.95 %
8 [Pcsclite-muscle] Lenovo laptops, GNU/Linux and smart card readers? 3 3.95 %
9 [Pcsclite-muscle] Chrome gets stuck and firefox won't open 3 3.95 %
10 [Pcsclite-muscle] Confused about extended APDU support 2 2.63 %
11 Potential security issues in CCID 2 2.63 %
12 [Pcsclite-muscle] SCardConnect behavior with invalid contexts 2 2.63 %
13 [Pcsclite-muscle] Missing checks of ATRDecodeAtr returns 2 2.63 %
14 [Pcsclite-muscle] Race condition during readerstate update 2 2.63 %
15 [Newsletter] Re: [Pcsclite-muscle] Race condition during 2 2.63 %
16 [Pcsclite-muscle] Alcor Micro AU9520 not recognized by libccid 2 2.63 %
17 New version of pcsc-lite: 1.8.26 1 1.32 %
18 [Pcsclite-muscle] Linux Kernel 5.5 / CCID / PCSCLITE change 1 1.32 %
19 [Pcsclite-muscle] pcsclite-muscle Digest, Vol 24, Issue 2 1 1.32 %
20 [Pcsclite-muscle] [Question] How to read SCARD_READER_CAPABILITIES 1 1.32 %
21 [Pcsclite-muscle] multi threading from a single pcsc handle / 1 1.32 %
22 [Pcsclite-muscle] Interfacing 32 bit driver with 64 pcscd 1 1.32 %
23 Unicode characters in a reader name 1 1.32 %
24 New version of libccid: 1.4.33 1 1.32 %
25 [Pcsclite-muscle] How to dectect card status? 1 1.32 %
26 [Pcsclite-muscle] Potential hang in SCardTransmit 1 1.32 %
27 [Pcsclite-muscle] Instances of Undefined behavior in CCID 1 1.32 %
28 Incomplete archive between May 2018 and June 2020 1 1.32 %
29 Lenovo laptops, GNU/Linux and smart card readers? 1 1.32 %
30 [Pcsclite-muscle] Race condition during readerstate 1 1.32 %
  other 6 7.89 %

Most used email clients:

   Mailer   Msg   Percent 
1 (unknown) 66 86.84 %
2 Mozilla/5.x 5 6.58 %
3 Evolution 3.36.5 (3.36.5-1.fc32) 3 3.95 %
4 K-9 Mail for Android 1 1.32 %
  other 1 1.32 %

Table of maximal quoting:

   Author   Percent 
1 mauro@faresoftware.it 19.95 %
2 jonathan.verner@nexusgroup.com 19.71 %
3 axel.braun@gmx.de 17.27 %
4 Axel.braun@gmx.de 11.83 %
5 t-pcsc@girst.at 11.03 %
6 tristan.degroof@multiversum.broedersvanliefde.be 10.60 %
7 suffsuccotash@gmail.com 9.36 %
8 thomas@schlenkhoff.me 7.59 %
9 gregor.waltz@raritan.com 6.58 %
10 stephan.guilloux@crisalid.com 4.97 %
11 ludovic.rousseau@gmail.com 3.57 %
12 trenta.sis@gmail.com 3.04 %
13 Marc.Kewitz.ext@rohde-schwarz.com 2.29 %
14 zeroconf@zeroconf.ee 1.75 %
15 godfreyhkchung@gmail.com 1.51 %
16 gsvelto@mozilla.com 0.43 %
17 emaxx@google.com 0.00 %
18 izh1979@gmail.com 0.00 %
19 sebastien@lorquet.fr 0.00 %
20 Gabriele Svelto 0.00 %
21 mstjohns@comcast.net 0.00 %
  average 3.59 %

Maximal quoting:

Author : ludovic.rousseau@gmail.com
Subject : [Pcsclite-muscle] Alcor Micro AU9520 not recognized by libccid 1.4.31
Date : Thu, 5 Nov 2020 14:43:12 +0100
Quote ratio: 62.97% / 18890 bytes

Longest message:

Author : ludovic.rousseau@gmail.com
Subject : [Pcsclite-muscle] Alcor Micro AU9520 not recognized by libccid 1.4.31
Date : Wed, 4 Nov 2020 22:37:47 +0100
Size : 729365 bytes

Most successful subject:

Subject : [Pcsclite-muscle] Workaround for missing suspend/resume
No. of msgs: 9
Total size : 90764 bytes

Final summary:

Total number of messages: 76
Total number of different authors: 21
Total number of different subjects: 35
Total size of messages (w/o headers): 1407974 bytes
Average size of a message: 18525 bytes

Happy new year 2021

 Dear readers,

I wish you a happy new year for 2021.

In 2020 I published 43 articles on this blog (25 in 2019, 24 in 2018, 34 in 2017 and 49 in 2016 and 51 in 2015). It is a 72% increase compared to 2019.

Audience

The number of readers in 2020 is decreasing (-15%) compared to 2019.


We still have the same 5 top countries in the same order than in 2019. USA is first one by a large margin.

 

I have a lot of readers (40%) coming from a Windows operating system.

It is surprising because I do not have many articles specifically about Windows. But at the same time a lot of my source codes or projects are also working on Windows.


Most read articles

No real surprise here: macOS & smart card and sample codes in different programming languages.

At the 8th place we find the article "pcsc_scan on Windows" I wrote in 2017. This article was not in the top-10 last year.


Thank you

Thank you to you, readers.

This blog has no advertising. If you want to support me you can send me some bitcoins or become a github sponsor.

PKCS#11 support in socat

socat

socat - Multipurpose relay is described upstream as:
Abstract
what:
"netcat++" (extended design, new implementation)
OS:
AIX, BSD, HP-UX, Linux, Solaris e.a. (UNIX)
lic:
GPL2
inst:
tar x...; ./configure; make; make install
doc:
README; socat.html, socat.1; xio.help
ui:
command line
exa:
socat TCP6-LISTEN:8080,reuseaddr,fork PROXY:proxy:www.domain.com:80
keyw:
tcp, udp, ipv6, raw ip, unix-socket, pty, pipe, listen, socks4, socks4a, proxy-connect, ssl-client, filedescriptor, readline, stdio, exec, system, file, open, tail -f, termios, setsockopt, chroot, fork, perm, owner, trace, dump, dgram, ext3, resolver, datagram, multicast, broadcast, interface, socket, sctp, generic, ioctl

The main idea of socat is to make a network connection between 2 hosts. It is possible to use a TLS connection. socat can use a private key stored in a file on disk. But it was not possible to use PKCS#11 as the cryptographic engine.

The benefit of using a PKCS#11 engine is that any PKCS#11 library can be used. So, of course, in my case the PKCS#11 library will be to use a smart card. So the private key will be inside a smart card and will stay protected inside the smart card.

My patch

socat already uses OpenSSL to process the private key.

OpenSSL already supports PKCS#11 through an "engine" mechanism.

It was rather easy to add support of OpenSSL engines in socat. I just replaced a call to SSL_CTX_use_PrivateKey_file() by a call to ENGINE_load_private_key() with the correct engine initialisation.
If the private key "filename" starts with "pkcs11:" then this is not a filename but a PKCS#11 URI scheme (defined in RFC-7512) already understood by OpenSSL pkcs11 engine.

The socat argument is then something like "key=pkcs11:id=%56%78;pin-value=1234"

diff --git a/sslcls.c b/sslcls.c
index f9ce389..ddcefd7 100644
--- a/sslcls.c
+++ b/sslcls.c
@@ -21,6 +21,8 @@
 #include "sysutils.h"
 #include "sycls.h"
 
+#include <openssl/engine.h>
+
 void sycSSL_load_error_strings(void) {
    Debug("SSL_load_error_strings()");
    SSL_load_error_strings();
@@ -214,10 +216,76 @@ int sycSSL_CTX_use_certificate_chain_file(SSL_CTX *ctx, const char *file) {
    return result;
 }
 
+static void display_openssl_errors(int l)
+{
+   const char *file;
+   char buf[120];
+   int e, line;
+
+   if (ERR_peek_error() == 0)
+       return;
+   Error2("At %s:%d:\n", __FILE__, l);
+
+   while ((e = ERR_get_error_line(&file, &line))) {
+       ERR_error_string(e, buf);
+       Error3("- SSL %s: %s:%d\n", buf, file, line);
+   }
+}
+
 int sycSSL_CTX_use_PrivateKey_file(SSL_CTX *ctx, const char *file, int type) {
    int result;
+
    Debug3("SSL_CTX_use_PrivateKey_file(%p, \"%s\", %d)", ctx, file, type);
-   result = SSL_CTX_use_PrivateKey_file(ctx, file, type);
+
+   if (0 == strncmp(file, "pkcs11:", 7))
+   {
+      /* Starts with "pkcs11:" -> use pkcs11 OpenSSL engine */
+      /* See RFC-7512: The PKCS #11 URI Scheme */
+      const char *privkey = file;
+      ENGINE *engine;
+      EVP_PKEY *pkey;
+      result = -1; /* error by default */
+
+      ENGINE_add_conf_module();
+
+      ENGINE_load_builtin_engines();
+
+      engine = ENGINE_by_id("pkcs11");
+      if (engine == NULL) {
+          Error("Could not get engine\n");
+          display_openssl_errors(__LINE__);
+          goto end;
+      }
+
+#if 0
+      if (!ENGINE_ctrl_cmd_string(engine, "VERBOSE", NULL, 0)) {
+          display_openssl_errors(__LINE__);
+          goto end;
+      }
+#endif
+
+      if (!ENGINE_init(engine)) {
+          Error("Could not initialize engine\n");
+          display_openssl_errors(__LINE__);
+          goto end;
+      }
+
+      pkey = ENGINE_load_private_key(engine, privkey, 0, 0);
+
+      if (pkey == NULL) {
+          printf("Could not load key\n");
+          display_openssl_errors(__LINE__);
+          goto end;
+      }
+      else
+          result = 1;
+
+end:
+      ENGINE_finish(engine);
+   }
+   else
+      result = SSL_CTX_use_PrivateKey_file(ctx, file, type);
+
    Debug1("SSL_CTX_use_PrivateKey_file() -> %d", result);
    return result;
 }


Usage

You need to have a certificate. For the example I will use OpenSSL to generate a self-signed certificate for the client but you can also use your own certification authority.

See socat documentation "Securing Traffic Between two Socat Instances Using SSL" for details.

Client side

Generate an RSA key pair:
openssl genrsa -out client.key 2048

Generate a self-signed certificate:
openssl req -new -key client.key -x509 -days 365 -out client.pem

Define some shell variables:
ID="ABCDEF"
INDEX=0
PIN=1234

Install the SoftHSM software PKCS#11 lib:

On Debian it is the softhsm2 package.
Of course you can use any PKCS#11 library and use a real smart card. But for now we will use a software only PKCS#11 lib for ease of use.

Create a new SoftHSM token
softhsm2-util --init-token --label "A token" \
    --pin $PIN --so-pin 123456 \
    --slot $INDEX

Install OpenSC to get pkcs11-tool command:
On Debian it is the opensc package.

Import the certificate:
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so  \
    --slot-index $INDEX --label "Certificate" \
    --write-object client.pem --type cert --id $ID

Import the private key:
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --slot-index $INDEX --label "Private key" \
    --write-object client.key --type privkey --id $ID --pin $PIN

List the imported objects:
pkcs11-tool --module /usr/lib/softhsm/libsofthsm2.so \
    --list-objects --pin $PIN
We get:
Using slot 0 with a present token (0x3a608bb2)
Private Key Object; RSA 
  label:      Private key
  ID:         abcdef
  Usage:      decrypt, sign, unwrap
  Access:     sensitive
Certificate Object; type = X.509 cert
  label:      Certificate
  subject:    DN: C=FR, ST=Some-State, L=Paris, O=Ludovic Rousseau blog, CN=test.example.org/emailAddress=foo@example.org
  ID:         abcdef

Delete the client private key since the key is now in the PKCS#11 token:
rm client.key

Install the OpenSSL PKCS#11 engine:
On Debian it is the package libengine-pkcs11-openssl that provides the file /usr/lib/x86_64-linux-gnu/engines-1.1/libpkcs11.so (on a x86_64 architecture).

Server side

Generate an RSA key pair:
openssl genrsa -out server.key 2048

Generate a self-signed certificate:
openssl req -new -key server.key -x509 -days 365 -out server.pem

Copy the client certificate file client.pem to the server, and the server certificate file server.pem to the client.

Run the socat server:
socat openssl-listen:4433,reuseaddr,cert=server.pem,cafile=client.pem,key=server.key -

Client side

Configure the OpenSSL engine:

Create a file named engine.conf and containing:

openssl_conf = openssl_init

[openssl_init]
engines = engine_section

[engine_section]
pkcs11 = pkcs11_section

[pkcs11_section]
engine_id = pkcs11
dynamic_path = /usr/lib/x86_64-linux-gnu/engines-1.1/libpkcs11.so
MODULE_PATH = /usr/lib/softhsm/libsofthsm2.so
init = 0

Define and export OPENSSL_CONF environment variable:

export OPENSSL_CONF=engine.conf

Run the client:
socat - 'openssl-connect:www.example.com:4433,cafile=server.pem,cert=client.pem,verify=1,key=pkcs11:id=%AB%CD%EF;pin-value=1234;token=A%20token'

Note the key=pkcs11:id=%AB%CD%EF;pin-value=1234 arguments.

Results

On the client I get:

2020/12/16 22:12:51 socat[6335] E SSL_connect(): error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure

On the server I get:

2020/12/16 22:12:52 socat[31137] E SSL_accept(): error:1417C0C7:SSL routines:tls_process_client_certificate:peer did not return a certificate

I have a problem with my self signed certificates. I searched for a few hours but without finding the solution.

Plan B: disable certificate checks

To make the server accept the client connection I have to add the socat argument verify=0 on the server side.

socat openssl-listen:4433,reuseaddr,cert=server.pem,cafile=client.pem,key=server.key,verify=0 -

Now the connection works fine in both direction. It is a bad solution. Don't do that on a production server.

Upstream integration

I sent my patch to the upstream maintainer, Gerhard Rieger, in Feb 2020. Gerhard will review the patch and give it a try.

Conclusion

I have no news from the socat maintainer since Feb 2020. So I decided to write my blog article even if my patch has not been reviewed and accepted or rejected.

May my patch help you secure a socat connection with your smart card (or another PKCS#11 token).

macOS Big Sur and smart cards status

macOS Big Sur (macOS 11.0) is now available since November, 2020.


tokend

A tokend is a piece of software used to bridge a cryptographic device (like a smart card) and the CDSA (Common Data Security Architecture) architecture.

Since macOS Lion (10.7 in 2011) the CDSA/tokend technology is deprecated. See "Mac OS X Lion and tokend".

tokend was disabled by default in Catalina but it was still possible to enable it again.

With macOS Big Sur tokend is now completely removed. The manpage SmartCardServices-legacy(7) is also no more present.

PC/SC

Since Yosemite (macOS 10.10 in 2014) the PC/SC layer is no more a fork of pcsc-lite. So comparing versions with pcsc-lite is useless.
% cat /System/Library/Frameworks/PCSC.framework/Versions/A/Resources/version.plist
<?xml
version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>BuildAliasOf</key> <string>CryptoTokenKit</string> <key>BuildVersion</key> <string>25</string> <key>CFBundleShortVersionString</key> <string>8.0</string> <key>CFBundleVersion</key> <string>1</string> <key>ProjectName</key> <string>SmartCardServices</string> <key>SourceVersion</key> <string>487040010000000</string> </dict> </plist>
The CFBundleShortVersionString is still 8.0 as for Mojave and Catalina. The SourceVersion changed from 408011002000000 to 487040010000000. But I have no idea what that means :-).
 
I have not yet made many tests of the PC/SC layer. So far it works fine.

Crypto Token Kit

CryptoTokenKit is the native smart card API since the complete rewrite in macOS Yosemite 10.10 (OS X Yosemite BETA and smart cards status).

The directory /System/Library/Frameworks/CryptoTokenKit.framework/CryptoTokenKit/ changed a bit between Catalina and Big Sur. For example the file CryptoTokenKit is no more present.

I tried my Objective-C sample and the code still works fine (as expected) even if the binary is now linked to a non-existent library file.
% otool -L ./blog.app/Contents/MacOS/blog
./blog.app/Contents/MacOS/blog:
	/System/Library/Frameworks/Foundation.framework/Versions/C/Foundation (compatibility version 300.0.0, current version 1673.126.0)
	/usr/lib/libobjc.A.dylib (compatibility version 1.0.0, current version 228.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1281.0.0)
	/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1673.126.0)
	/System/Library/Frameworks/CryptoTokenKit.framework/Versions/A/CryptoTokenKit (compatibility version 1.0.0, current version 1.0.0)
% ls /System/Library/Frameworks/CryptoTokenKit.framework/Versions/A/CryptoTokenKit
ls: /System/Library/Frameworks/CryptoTokenKit.framework/Versions/A/CryptoTokenKit: No such file or directory

CCID

% grep -A 1 CFBundleShortVersionString /usr/libexec/SmartCardServices/drivers/ifd-ccid.bundle/Contents/Info.plist
	<key>CFBundleShortVersionString</key>
	<string>1.4.32</string>
Apple updated the CCID driver from version 1.4.31 in Catalina to 1.4.32 in Big Sur.

Version 1.4.32 is not the latest version available. I released this version on April, 22th 2020.
The latest version (for now) of the CCID driver is 1.4.33 released on June 25th, 2020. 

Apple Silicon

macOS Big Sur is also the operating system for the new Apple computers using the Apple Silicon CPU (an ARM based CPU). The binaries provided with macOS Big Sur are now also compiled for ARM.

For example with the CCID driver:
% cd /usr/libexec/SmartCardServices/drivers/ifd-ccid.bundle/Contents/MacOS/
% file libccid.dylib 
libccid.dylib: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit dynamically linked shared library x86_64] [arm64e:Mach-O 64-bit dynamically linked shared library arm64e]
libccid.dylib (for architecture x86_64):	Mach-O 64-bit dynamically linked shared library x86_64
libccid.dylib (for architecture arm64e):	Mach-O 64-bit dynamically linked shared library arm64e
My CCID driver works fine with GNU/Linux on a RaspberryPi with an ARM CPU. So it is not surprising that it works also fine with an Apple Silicon CPU.

When Apple will publish the patches they made to Free Software programs used in Big Sur at https://opensource.apple.com we will see if some modifications were needed.

Conclusion

No big changes in Big Sur for the smart card world.