Introduction
Recently I got some patches introducing the crypto engine framework merged into the crypto layer for v4.6, which are applied in Herbert Xu’s git:  http://git.kernel.org/cgit/linux/kernel/git/herbert/cryptodev-2.6.git

Background
In crypto layer, the block cipher hardware engine driver fetches the request from dm-crypt into its queue to wait for its thread or workqueue to handle, and finalizes the request when finishing the encryption/decryption.

However, the old method needed each hardware engine driver to implement and maintain their own queue and thread for processing requests.  Previously the API provided helpers for only the queue itself (in crypto_enqueue_request() and crypto_dequeue_request()) but they don’t help with the mechanics of driving the hardware (things like running the request immediately, DMA map it or providing a thread to process the queue in), even though a lot of that code really shouldn’t vary that much from device to device.

Introduce crypto engine framework
Therefore we provide a crypto engine framework that implements the queue and thread for pushing requests to the hardware, as the hardware becomes free so that drivers could use it. At the same time it can avoid some reduplicated code in hardware engine driver.

How to use it
The section below shows how to integrate with the new crypto engine framework to make the omap aes driver be under utilized.

(1) Remove the request queue and the thread/workqueue/tasklet which is used to queue requests in your driver, meanwhile add the ‘crypto_engine’ structure in your device descriptor structure.


@@ -152,13 +153,10 @@ struct omap_aes_dev {
unsigned long           flags;
int                     err;
-       spinlock_t              lock;
-       struct crypto_queue     queue;
-
struct tasklet_struct   done_task;
-       struct tasklet_struct   queue_task;
struct ablkcipher_request       *req;
+       struct crypto_engine            *engine;
@@ -1177,9 +1160,6 @@ static int omap_aes_probe(struct platform_device *pdev)
-       spin_lock_init(&dd->lock);
-       crypto_init_queue(&dd->queue, OMAP_AES_QUEUE_LENGTH);
@@ -1211,7 +1191,6 @@ static int omap_aes_probe(struct platform_device *pdev)
tasklet_init(&dd->done_task, omap_aes_done_task, (unsigned long)dd);
-       tasklet_init(&dd->queue_task, omap_aes_queue_task, (unsigned long)dd);


(2) Add some code to initialize the crypto engine.
You can initialize the crypto engine with crypto_engine_alloc_init() function and remove it with crypto_engine_exit() function.

After the initialization for crypto engine, now you can start the crypto engine by calling crypto_engine_start(). If there are some errors or other problem during queuing requests, you can stop the engine by crypto_engine_stop() function.

Usually drivers need to provide a ‘prepare_crypt_hardware’ function to prepare the hardware when the hardware becomes active and an‘unprepare_crypt_hardware’ function to disable the hardware when there are no requests to do.

Drivers can also set ‘prepare_request’ member if they need do some preparation before handling the current request and undo it by setting ‘prepare_request’ when finalizing one request.

Drivers must also provide a function to handle each request by setting ‘crypt_one_request’.


@@ -1252,7 +1231,20 @@ static int omap_aes_probe(struct platform_device *pdev)
+       /* Initialize crypto engine */
+       dd->engine = crypto_engine_alloc_init(dev, 1);
+       if (!dd->engine)
+               goto err_algs;
+
+       dd->engine->prepare_request = omap_aes_prepare_req;
+       dd->engine->crypt_one_request = omap_aes_crypt_req;
+       err = crypto_engine_start(dd->engine);
+       if (err)
+               goto err_engine;
+
return 0;
+err_engine:
+       crypto_engine_exit(dd->engine);
@@ -1288,8 +1279,8 @@ static int omap_aes_remove(struct platform_device *pdev)
+       crypto_engine_exit(dd->engine);


(3) Implement callbacks to prepare requests and encrypt one request.


+static int omap_aes_prepare_req(struct crypto_engine *engine,
+                               struct ablkcipher_request *req)
+{
+       struct omap_aes_ctx *ctx = crypto_ablkcipher_ctx(
+                       crypto_ablkcipher_reqtfm(req));
+       struct omap_aes_dev *dd = omap_aes_find_dev(ctx);
+       struct omap_aes_reqctx *rctx;
+       int len;
-       req = ablkcipher_request_cast(async_req);
+       if (!dd)
+               return -ENODEV;
dd->req = req;
+       return omap_aes_write_ctrl(dd);
+}
+static int omap_aes_crypt_req(struct crypto_engine *engine,
+                             struct ablkcipher_request *req)
+{
+       struct omap_aes_ctx *ctx = crypto_ablkcipher_ctx(
+                       crypto_ablkcipher_reqtfm(req));
+       struct omap_aes_dev *dd = omap_aes_find_dev(ctx);
+
+       if (!dd)
+               return -ENODEV;
+
+       return omap_aes_crypt_dma_start(dd);
}


(4) Modify the handle queue method replacing with crypto_transfer_request_to_engine() function.
You can issue crypto_transfer_request_to_engine() function to add a request into the engine queue and wait for pushing the request by engine.


static int omap_aes_handle_queue(struct omap_aes_dev *dd,
struct ablkcipher_request *req)
{
-       struct crypto_async_request *async_req, *backlog;
-       struct omap_aes_ctx *ctx;
-       struct omap_aes_reqctx *rctx;
-       unsigned long flags;
-       int err, ret = 0, len;
-
-       spin_lock_irqsave(&dd->lock, flags);
if (req)
-               ret = ablkcipher_enqueue_request(&dd->queue, req);
-       if (dd->flags & FLAGS_BUSY) {
-               spin_unlock_irqrestore(&dd->lock, flags);
-               return ret;
-       }
-       backlog = crypto_get_backlog(&dd->queue);
-       async_req = crypto_dequeue_request(&dd->queue);
-       if (async_req)
-               dd->flags |= FLAGS_BUSY;
-       spin_unlock_irqrestore(&dd->lock, flags);
+               return crypto_transfer_request_to_engine(dd->engine, req);
-       if (!async_req)
-               return ret;
+       return 0;
+}


(5) Modify the method to finalize one request replacing with crypto_finalize_request() function.
When one request is finished encrypting or decrypting, you can finalize the request by crypto_finalize_request() function.


@@ -532,9 +530,7 @@ static void omap_aes_finish_req(struct omap_aes_dev *dd, int err)
-       req->base.complete(&req->base, err);
+       crypto_finalize_request(dd->engine, req, err);
}


Once all above modifications are completed, the crypto engine framework can work well on your driver.

In the future

Current problem
Users will use dm-crypt, which is a transparent disk encryption, to encrypt and decrypt block device. Refer to the link to see what is dm-crypt (https://en.wikipedia.org/wiki/Dm-crypt).

Now dm-crypt will send encryption or decryption requests to the crypto layer one block at a time, making each request 512 bytes long, which is a much smaller size for hardware engine, which means the hardware engine cannot deliver its best performance.

Introduce bulk mode
Some cipher hardware engines prefer to handle bulk block rather than one sector (512 bytes) created by dm-crypt, cause these cipher engines can handle the intermediate values (IV) by themselves in one bulk block. This means we can increase the size of the request rather than always 512 bytes and thus increase the hardware engine processing speed, which is the bulk mode for crypto engine framework what we want to introduce in the future.

There are more works that need to be investigated in the future, especially in below 3 problems:
(1) Finding the best method to check the requests are compatible for merging requests.
(2) Thinking of the request numbers on how much we were able to combine together for bulk mode.
(3) How much benefit we can get from bulk mode.

“Sir, it’s very possible this asteroid is not stable.” ~ C3PO

While PCIe passthrough (the process of assigning a PCIe device to a VM, also known as device assignment) is supported through a mostly architecture-agnostic subsystem called VFIO, there are intricate details of an ARM-based system that require special support for Message Signaled Interrupts (MSIs) in the context of VFIO passthrough on ARM server systems.

Message Signaled Interrupts

MSIs are an alternative to wire based interrupts. A device using MSIs does not need a dedicated line to the interrupt controller. Instead, to trigger interrupts a device simply writes at a specific memory address belonging to a piece of HW that can generate interrupts as a result of the memory write.  Such hardware is typically referred to as an MSI controller. The MSI controller derives an interrupt ID from the written message.

Thus, the MSI-enabled device must be programmed with:

• the address to write to
• and a payload

ARM MSI Controllers

This chapter gives a brief introduction of 2 ARM MSI controllers, the GICv2m and the GICv3 ITS. We purposely present a simplified overview. Please refer to the references for more details.

GICv2M

The GICv2m widget contains one or more MSI frames. Each MSI frame is wired up to a set of GIC SPI wires (shared peripheral interrupt). MSI frames should not target the same SPI IDs for isolation purpose.

From the CPU perspective each MSI frame is 4kB wide and contains some info registers telling the base and number of associated SPI IDs and the MSI_SETSPI_NS 32-bit register.

The MSI_SETSPI_NS is also referred to as the doorbell. Writes to this register trigger SPI to the GIC. The data payload allows to select the triggered SPI ID.

The GICv2M does not require any functional programming since the SPIs are statically assigned to each MSI frame.

Separate MSI frames are provisioned for interrupt isolation purpose. Each frame is supposed to target separate SPI ID windows. Devices attached to separate MSI frames have different SPI ID domains. Of course MSI frame access must be restricted by the bus topology, an IOMMU, or by other means. On the other hand, a system with a single MSI frame cannot do HW IRQ isolation between devices allowed to access that single MSI frame.

GICv3 ITS

GICv3 supports a compatibility mode where a similar mechanism as GICv2m is used. But more importantly it supports the Interrupt Translation Service (ITS) mechanism. The ITS exposes a single 64kB MSI frame. This MSI frame contains the GITS_TRANSLATOR register (ITS doorbell register). This is the address to be written when a device wants to trigger an interrupt. The ITS implements a translation mechanism that takes as input the eventid passed in the MSI data payload, a device id (conveyed out-of-band, typically on the AXI user bits) and outputs an LPI id. LPI stands for Local Peripheral Interrupt. The GIC HW takes this LPI ID as input.

As opposed to the GICv2M, the ITS must be configured by software before it is used. For example, translation tables need to be programmed before any MSI translation can succeed:

• A device table entry must exists per deviceid, pointing to a device interrupt translation table
• An entry must exist in the device interrupt translation table for each eventid the device is likely to produce. This entry basically tells which LPI ID to trigger (and the CPU it targets)

Interrupt translation is also supported on Intel hardware as part of the VT-d spec. The Intel IRQ remapping HW provides a translation service similar to the ITS. The difference is that the intel implementation looks more like a true IOMMU in the sense the translation process uses the MSI address as input as well the MSI data payload. On ARM the deviceid is conveyed out of band.

So on x86 there is not a single doorbell address MSI messages are written to. Instead each device writes at a different address. This address is within  the 0xFEEX_XXXXh range and bits 14:0 and bit 2 of the upper 32-bit of the address encode the deviceid (handle) used by the translation process.

For that reason the IRQ remapping HW is abstracted by IOMMU drivers.  On ARM, ITS is abstracted by irqchip driver.

KVM PCI/MSI passthrough, x86/ARM Differences

This chapter explains why the current VFIO integration (QEMU VFIO PCI device/ kernel VFIO PCI driver) does not work for ARM.

When a device is assigned to a guest, it is unbound from its native driver and bound to the VFIO-PCI driver. A prerequisite for using VFIO in full feature mode is to have an IOMMU downstream to the device. Indeed The VFIO driver API eventually allows the user-space to set up DMA mappings between the device IOVA space and user-space virtual memory. If no IOMMU mapping does exist for a given IOVA, the related DMA access fails with an IOMMU abort. IOMMU allows DMA access isolation.

In the virtualization use case, The QEMU VFIO device takes care of mapping all the guest ram region physical addresses to allow them to be accessed by the assigned device. However only the RAM regions are mapped, meaning the peripheral register spaces are not mapped.

On x86 this does not bring any issue since MSI write translation hit within a special 1MB physical address window [FEE0_0000h – FEF0_000h]. Those transactions target the APIC configuration space and not DRAM, meaning the downstream IOMMU is bypassed. So there is no need to IOMMU map the MSI transaction addresses.

On ARM however the MSI transactions towards the doorbell are conveyed through the IOMMU. Therefore an IOMMU mapping must exist. This is similar on PowerPC.

Without changes to the VFIO subsystem, MSIs simply cause IOMMU aborts because no mapping is defined between the address used by the device (IOVA) and the physical address for the MSI frame including the doorbell register.

The goal of the ongoing work is to create the needed IOMMU mappings for MSI write transactions to eventually reach the hardware MSI frame.

Assigned device MSI Setup

This chapter describes how an MSI is setup for an assigned device. First we discuss the VFIO legacy implementation (upstreamed implementation working for x86). Then we explain the adaptations needed to make it functional on ARM/ARM64.

Legacy Implementation

VFIO decouples the MSI configuration of the physical PCIe device from the configuration performed by the guest driver.

Practically the MSI message (address/data) programmed by the guest are not used to program the actual physical PCIe device. PCIe configuration space accesses performed by the guest are trapped by VFIO/KVM. The MSI address never is used. The data payload computed by the guest matches a virtual SPI ID and not a physical SPI ID.

Instead, when the user-space sets IRQ signalling up (VFIO_DEVICE_SET_IRQ ioctl), the host VFIO PCI driver retrieves MSI vectors from the host MSI sub-system. Therefore, it programs the assigned PCIe devices with an MSI message composed by the host msi-parent MSI controller.

Then the MSI forwarding follows that path:

host computed MSI message -> host computed physical SPI ID -> eventfd -> guest computed virtual SPI ID

Requested adaptation for ARM

When the VFIO PCI driver programs the assigned physical PCIe device with an MSI message composed by the host, we need to replace the MSI message address (the doorbell host physical address) by an IOVA, mapped onto this doorbell physical address.

So we need to identify an IOVA that is not already used (ie. not an IOVA matching any guest RAM region GPA). The choice, then, is to either (1) use an IOVA known to the guest (in its GPA space), for example belonging to a virtual GICv2m or other MSI controller created and presented to the guest, or (2) just use some other IOVA, not corresponding to any RAM region.

It feels natural that user-space communicates the address of a virtual MSI controller to the guest (GICv2m single MSI frame on our case). However we saw the virtual side of MSIs and physical side are completely decoupled. Also a device may need multiple MSI frames or the host MSI controller may be different in character from the virtual one given to the guest.

So we currently choose not to use the GICv2m MSI frame GPA. Instead QEMU provides a pool of unused GPA to VFIO.

In mach-virt we have a platform bus which represents a pool of IRQ and MMIO pages. The platform bus is currently used for dynamic instantiation of sysbus devices, especially VFIO platform devices. This 32MB GPA pool with its own GPA allocator is quite well suited to provide an anonymous contiguous pool of GPA=IOVA usable to map MSI frames. This integration does not induce any change in mach-virt memory map.

We reuse the VFIO DMA MAP ioctl to pass this reserved IOVA region. A new flag (VFIO_DMA_FLAG_MSI_RESERVED_IOVA ) is introduced to differentiate such reserved IOVA from RAM IOVA. Then the base/size of the window is passed to the IOMMU driver though a new function introduced in the IOMMU API.

The IOVA allocation within the supplied reserved IOVA window is performed on-demand, when the MSI controller composes/writes the MSI message in the PCIe device. Also the IOMMU mapping between the newly allocated IOVA and the backdoor address page is done at that time. The MSI controller uses a new function introduced in the IOMMU API to allocate the IOVA and create an IOMMU mapping.

So there are adaptations needed at VFIO, IOMMU and MSI controller level. The extension of the IOMMU API still is under discussion. Also changes at MSI controller level need to be consolidated.

Interrupt Safety

When an MSI enabled device is assigned to a guest we need to guarantee it cannot trigger MSIs that correspond to interrupt IDs of devices belonging to the host or other guests. Indeed once a device gets access to an MSI frame, shared with others, nothing prevents a malicious user-space driver to trigger DMA requests within that region. This can lead to denial of service attacks.

On the figure below we can image device #0 is used by the host while devices #1 and #2 are assigned to a guest.

Interrupt Safety with GICv2m

On GICv2m the standard way to implement interrupt isolation is to support several MSI frames and make sure guests are assigned with separate MSI frames (host and each guest must have separate MSI frames). This is due to the fact the GICv2m does not support interrupt translation, also known as IRQ remapping (Intel naming).

Also even with multiple MSI frames, an SR-IOV PCI device attached to a PCI host controller would have a single MSI parent frame. We could not have one VF (virtual function) assigned to one guest and another assigned to another guest for security reasons.

Since the HW does not support IRQ remapping, the host kernel would need to check that devices attached to a VFIO group do not share an MSI frame with devices outside of the group. Performing such a check in software involves extending the VFIO notion of group viability. This would bring a significant design complexity and the choice was made to consider MSI passthrough without IRQ remapping capable msi-parent as unsafe.

If the end-user takes the risk to enable such passthrough, he must explicitly load the VFIO_IOMMU_TYPE1 module with allow_unsafe_interrupts parameter set to 1 (see the User Perspective section). This is an obvious limitation, but works the same way in the x86 world.

Interrupt Safety with GICv3 ITS

With GICv3 ITS we do not have this issue since each MSI transaction is tagged with a device-id and the device-id makes possible to separate the LPI domains. The ITS supports IRQ remapping similarly to the Intel VT-d IRQ remapping HW: MSI passthrough is safely supported and users do not need to use the allow_unsafe_interrupts parameter.

Conclusions

Supporting MSI passthrough with KVM on ARM platforms requires changes to Linux and QEMU due to underlying differences between the ARM and x86 architectures. ARM platforms with GICv2m MSI controllers will require users to load VFIO with the allow_unsafe_interrupts parameter for MSI passthrough to work, but GICv3 ITS platforms will work with VFIO without any additional parameters.

The changes required to Linux and QEMU are currently being upstreamed by Linaro and the latest versions of the patch series are referenced below [5, 6].

User Perspective

This chapter illustrates the assignment of 2 different PCIe devices:

• Intel 82574L Ethernet Controller (e1000e)
• Intel X540-T2 Ethernet Controller (SR-IOV capable)

on  AMD 64-bit ARM Overdrive featuring a single GICv2M MSI frame.

e1000e Assignment

Host Compilation

make defconfig
scripts/config -e CONFIG_IOMMU_SUPPORT
scripts/config -e CONFIG_IOMMU_API
scripts/config -e CONFIG_ARM_SMMU
scripts/config -m CONFIG_VFIO
scripts/config -m CONFIG_VFIO_PCI
scripts/config -m CONFIG_VFIO_IOMMU_TYPE1
scripts/config -e CONFIG_NETDEVICES
scripts/config -e CONFIG_NET_VENDOR_AMD
scripts/config -e CONFIG_AMD_XGBE
scripts/config -e CONFIG_E1000E

Host PCIe Topology

00:00.0 0600: 1022:1a00
Subsystem: 1022:1a00
00:02.0 0600: 1022:1a01
00:02.2 0604: 1022:1a02
Kernel driver in use: pcieport
01:00.0 0200: 8086:10d3
      Subsystem: 8086:a01f
      Kernel driver in use: e1000e

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1a00
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1a01
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1a02
01:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

Module Loading

allow_unsage_interrupts opt-in:
sudo modprobe -v vfio-pci
sudo modprobe -r vfio_iommu_type1
sudo modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1

VFIO-PCI driver binding

The following command lines unbind the native e1000e driver and bind the vfio-pci driver instead:

echo vfio-pci > /sys/bus/pci/devices/0000:01:00.0/driver_override
echo 0000:01:00.0 > /sys/bus/pci/drivers/e1000e/unbind
echo 0000:01:00.0 > /sys/bus/pci/drivers_probe

QEMU command line example

qemu-system-aarch64 -M virt -smp 4 -m 12G -cpu host -serial stdio -display none \
–enable-kvm -kernel /root/VM/Image \
-drive if=none,cache=writethrough,file=/root/VM/ubuntu10.img,format=raw,id=guestrootfs \
-device virtio-blk-device,drive=guestrootfs \
-net none \
-device vfio-pci,host=01:00.0 \
-append ‘loglevel=8 root=/dev/vda rw console=ttyAMA0 earlyprintk ip=dhcp’

X540-T2 (SR-IOV capable) Assignment

Host Compilation

ACS Capability Override
PCIe ACS capability (Access Control Service) is not properly exposed on this HW.

Without any action the PF (physical function) and all the VFs (virtual functions) belong to the same iommu group. This prevents from assigning the VF since the vfio group is not viable (the PF must remain bound to the ixgbe native driver else the VFs disappear).

This problem is pretty well know on other architectures too. There is a patch available to hack and workaround the issue but this one most probably will never been upstreamed: https://lkml.org/lkml/2013/5/30/513. At least it makes possible to experience SR-IOV passthrough.

After applying the patch and adding pcie_acs_override=downstream to the grub cmd line the PF/VF are in separate iommu groups.

ixgbe Module Addition

Compile the ixgbe as a module (needed to turn VFs on) by adding the following options:
scripts/config -m CONFIG_IXGB

scripts/config -m CONFIG_IXGBE
scripts/config -m CONFIG_IXGBEVF

Host PCIe Topology
Before SR-IOV enabling:
00:00.0 0600: 1022:1a00
       Subsystem: 1022:1a00
00:02.0 0600: 1022:1a01
00:02.2 0604: 1022:1a02
       Kernel driver in use: pcieport
01:00.0 0200: 8086:1528 (rev 01)
       Subsystem: 8086:0002
       Kernel driver in use: ixgbe

SR-IOV enabling:

reload the ixgbe module with max_vfs parameter set to the wished number of virtual functions:

modprobe -r ixgbe
modprobe ixgbe max_vfs=2

Now the PCIe topology looks like:

-[0000:00]-+-00.0
          +-02.0
          \-02.2-[01]–+-00.0
                       +-10.0
                       \-10.2

00:00.0 0600: 1022:1a00
       Subsystem: 1022:1a00
00:02.0 0600: 1022:1a01
00:02.2 0604: 1022:1a02
       Kernel driver in use: pcieport
01:00.0 0200: 8086:1528 (rev 01) eth4
       Subsystem: 8086:0002
       Kernel driver in use: ixgbe
01:10.0 0200: 8086:1515 (rev 01)
       Subsystem: 8086:0002
       Kernel driver in use: ixgbevf
01:10.2 0200: 8086:1515 (rev 01)
       Subsystem: 8086:0002
       Kernel driver in use: ixgbevf

Allow Unsafe Interrupts

sudo modprobe -v vfio-pci
sudo modprobe -r vfio_iommu_type1
sudo modprobe -v vfio_iommu_type1 allow_unsafe_interrupts=1

Physical Function Enable

The PF must be enabled before assigning the VFs.
ifconfig eth4 up

VFIO-PCI driver binding

unbind the ixgbevf native driver and bind vfio-pci driver instead:

echo vfio-pci > /sys/bus/pci/devices/0000:01:10.0/driver_override
echo 0000:01:10.0 > /sys/bus/pci/drivers/ixgbevf/unbind
echo 0000:01:10.0 > /sys/bus/pci/drivers_probe

QEMU Command line

qemu-system-aarch64 -M virt -smp 4 -m 4096 -cpu host -serial stdio -display none \
–enable-kvm -kernel /root/VM/Image1 \
-drive if=none,cache=writethrough,file=/root/VM/ubuntu10.img,format=raw,id=guestrootfs -device virtio-blk-device,drive=guestrootfs \
-net none -device vfio-pci,host=01:10.0 \
-append ‘loglevel=8 root=/dev/vda rw console=ttyAMA0 earlyprintk ip=dhcp’

References

Documents

Kernel & QEMU Series

“But it does move!” ~ Gaileo Galilei

“Digital circuits are made from analog parts.” ~ Don Vonada

Linaro Connect Bangkok (BKK16) was our 19th event to date and our first in Thailand. It was a great event at a beautiful venue. Our Thai hosts at the Centara Grand were wonderfully welcoming and helped to make BKK16 an event to remember.

On Monday 7 March,  George Grey, CEO of Linaro, started proceedings with his now traditional welcome keynote.  He made several announcements including the ARM Based Developer Cloud and 96Boards HiKey in AOSP.

There were also a number of exciting demos hosted,  which all focused on the 96Boards  platform including:

1. Tony Zhang from LeMaker demonstrating the first 96Boards CE LCD Display
2. Yang Zhang, 96Boards Director presenting a 96Boards Virtual Reality Demonstration
3. Martin Stadtler LEG Director providing a demonstration of a cloud based application owncloud, using nginx, memcached, haproxy, apache and mysql, all on ARM64 servers, running the enterprise reference software platform.
4. Jon Masters Chief Architect from Red Hat.

Keynotes

We were very excited to have had some fantastic keynote speakers that got a lot of engagement from our attendees. Security in a device connected world was a common theme each morning. The message was “Yes we should be very worried about security vulnerabilities in connected devices (light bulbs, thermostats, teddy bears…) and we have a responsibility as engineers to minimize these issues as much as humanly possible. “

• Monday 7 March – George Grey, Linaro CEO
  • Keynote video and slides: Welcome to BKK16
• Monday 7 March – Eric Hennenhoefer, VP ARM Research
  • Keynote video and slides: From Sensors to Supercomputers, Big Data begins with Little Data

• Tuesday 8 March – Matthew Garrett, Principal Security Developer at CoreOS
  • Keynote video and slides: Turning Specifications into Security
• Wednesday 9 March – Shane Coughlan, General Assembly Member of FS Foundation EU and Executive Director of OpenForum EU and Harald Welte of GPL Violations held a joint keynote
  • Keynote video and slides: OSS Compliance
• Thursday 10 March – Per Beming, Chief Architect Network Business Unit Radio
  • Keynote video and slides: The Networked Society and 5G
• Friday 11 March – David Rusling, Linaro CTO
  • Keynote video and slides: The End-to-End Story

In our next blog entry, we will highlight some of the most popular sessions at BKK16.

On this series:

• BKK16 Summary Blog 1 – Keynotes (This one)
• BKK16 Summary Blog 2 – Sessions
• BKK16 Summary Blog 3 – Recognition and Demo Friday

Follow us on social media:
Interviews on youtube, Keynote on youtube, Presentations (Slideshare),
Photos (Flickr), Facebook, Twitter and Linkedin

Next week at Linaro Connect LAS16 there will be a Firmware Summit on Tuesday September 27, 2016 from 10:10am-1:00pm (PST).  The purpose of this summit is to bring the key developers and maintainers from all camps in the same place, review the status and plan the next steps.

The Linaro Enterprise Group has been driving the work to implement, upstream and maintain UEFI and ACPI support on ARM platforms since its creation in 2012. Over the years, the team has initiated new activities, namely the OpenPlatformPkg proposal to support multiple SoCs and multiple platforms in EDK2, as well as supporting more and more features on ACPI, e.g. recently, Console Selection, APEI, PCIe, NUMA and new GIC implementations. In order to successfully deploy ARM servers in production, it is required that firmware (BIOS) engineers and kernel engineers work in close collaboration and drive the addition of new platforms, improved SoC support, etc.

Recently Linaro founded devicetree.org to support making device tree a properly managed and supported standard. This includes reducing fragmentation, improving maintainability and increasing multiplatform support in hardware description thanks to an open process that encourages wide community participation and the current best practices and technology.

Leif Lindholm, one of the speakers for the summit has just published a great blog on “UEFI Driver Development” that discusses writing a standalone driver from scratch.  His blog also discusses loading it from the UEFI Shell, detecting the presence of a device it recognizes and unloading it from the UEFI shell.  This is a great introduction related to some of the topics in the summit next week.   To read his blog please go to:  http://blog.eciton.net/uefi/uefi-driver-part1.html 

Below is an overview of the sessions that will be part of the summit:

Title:  ARM64 ASWG and Linux ACPI update
Abstract: As presented at previous connects an update on the current support for ARM64 at ASWG level and the hot topics. Also an update on the support that is mainline in the Linux kernel for ARM64 ACPI support. Also covering the next steps for both ASWG and Linux support. Followed by a discussion period.
Speaker: Al Stone, Hanjun Guo

Title: SCMI – System Management and Control Interface
Abstract: In this session we present a new standard proposal for system control and management. The industry, both in high end mobile and enterprise, is trending towards the use of power and system controllers. In most cases the controllers have very similar communication mechanisms between application processors and controllers. In addition, these controllers generally provide very similar functions, e.g. DVFS, power domain management, sensor management. This standard proposal provides an extensible, OS agnostic, and virtualizable interface to access these functions.
Speaker(s):Charles Garcia-Tobin

Title: Tianocore Progress and Status
Abstract:  A brief update on the progress of ARM64 support in EDK2 and OpenPlatformPkg. Also covering the next steps. Followed by a discussion period.
Speaker: Leif Lindholm

Title: Secure Boot
Abstract:  A 101 style introduction to what Secure Boot is as Secure means different things to different people. Covering the current status, what features are implemented currently on ARM64 and what features should be implemented in the future. Followed by a discussion period.
Speaker: Ard Biesheuvel

Title: RAS What is it? Why do we need it?
Abstract:  A 101 style introduction to RAS, its purpose and how we use it on ARM64. Covering current status of implementation in ASWG specs and Linux kernel. Plans for future features that are essential for ARM64. Followed by a discussion period.
Speaker: Yazen Ghannam, Fu Wei

To learn more about Linaro Connect LAS16 please visit:  http://connect.linaro.org/las16/

Next week at Linaro Connect LAS16 there will be a Microcontroller Software Summit on Wednesday September 28, 2016 from 10:10am-1:00pm (PST).  The summit looks to focus on few of the topics that the microcontroller and Cortex-M based SoC and systems are facing.  There are numerous software solutions in the microcontroller space and we look to bring together developers from the various RTOSes (Zephyr, Apache Newt, ARM mbed) to discuss solutions to common problems faced by everyone. The summit will focus on Device Configuration, Build Systems, and Security.

As the time to market continues to short the need the be able to get a board port going faster is increasingly important.  In addition, multiple software components now need to be aware of various board specific details and thus each software component has to be ported specifically to a given board/platform.  We are looking at solutions for describing the board, platform, system, SoC, pins, clocks, etc in a vendor neutral way that multiple software components can utilize as a source of configuration information.  The Device Configuration session will focus on a proposed solution to this problem, some of the issues we need to address, and some of the tooling areas that we need to work on to see this solution be a success.

Microcontroller systems in a connected world are doing more and more and thus we are pulling in software stacks from more and more places.  How we source, build and compose these software packages together to provide images for a production system is becoming extremely complicated.  In addition, we have unique requirements in the microcontroller space as there may not be a separation of OS from user space or a filesystem present.  The Build System session will focus on looking at what the unique requirements and solutions there are for how we compose all this software together.  We will look at some existing solutions from Apache MyNewt, to Open Embedded, as well as previous attempts by ARM mbed with yotta to address this problem.

Finally, as we connect devices to the internet, security is of paramount concern.  In the Cortex-A world we are use to technologies like TrustZone, encryption, public key authentication, signed software image, however these are new to the microcontroller world.  In addition the microcontroller systems are far more resourced constrained.  The session will look at some of the differences between the two classes of system and what type of unique requirement exist in the microcontroller world.  We hope to capture some of the unique requirements and discuss areas that need software standardization to ensure security throughout the system.

In addition to the Summit there are several IOT and embedded sessions that will go into more depth on particular RTOSes, software environments, and security topics relevant to the microcontroller space.

* LAS16-100: Zephyr Technical Overview

* LAS16-112 mbed OS Technical Overview

* LAS16-104: Apache MyNewt technical overview

* LAS16-108: JerryScript and other scripting languages for IoT

* LAS16-203: Platform security architecture for embedded devices

* LAS16-300K2: Geoff Thorpe – IoT Zephyr

* LAS16-407: Internet of Tiny Linux (IoTL): the sequel.

To Learn more about these sessions and Linaro Connect visit:  http://connect.linaro.org/las16/

Introduction

The Linux kernel supports a variety of sleep states.  These states provide power savings by placing the various parts of the system into low power modes.  The four sleep states are suspend to idle, power-on standby (standby), suspend to ram, and suspend to disk.  These are also referred to sometimes by their ACPI state: S0, S1, S3, and S4, respectively.  Suspend to idle is purely software driven and involves keeping the CPUs in their deepest idle state as much as possible.  Power-on standby involves placing devices in low power states and powering off all non-boot CPUs.  Suspend to ram takes this further by powering off all CPUs and putting the memory into self-refresh.  Lastly, suspend to disk gets the greatest power savings through powering off as much of the system as possible, including the memory.  The contents of memory are written to disk, and on resume this is read back into memory.

This blog post focuses on the implementation of suspend to idle.  As described above, suspend to idle is a software implemented sleep state.  The system goes through a normal platform suspend where it freezes the user space and puts peripherals into low-power states.  However, instead of powering off and hotplugging out CPUs, the system is quiesced and forced into an idle cpu state.  With peripherals in low power mode, no IRQs should occur, aside from wake related irqs.  These wake irqs could be timers set to wake the system (RTC, generic timers, etc), or other sources like power buttons, USB, and other peripherals.

During freeze, a special cpuidle function is called as processors enter idle.  This enter_freeze() function can be as simple as calling the cpuidle enter() function, or can be much more complex.  The complexity of the function is dependent on the SoCs requirements and methods for placing the SoC into lower power modes.

Prerequisites

Platform suspend_ops

Typically, to support S2I, a system must implement a platform_suspend_ops and provide at least minimal suspend support.  This meant filling in at least the valid() function in the platform_suspend_ops.  If suspend-to-idle and suspend-to-ram was to be supported, the suspend_valid_only_mem would be used for the valid function.

Recently, however, automatic support for S2I was added to the kernel.  Sudeep Holla proposed a change that would provide S2I support on systems without requiring the implementation of platform_suspend_ops.  This patch set was accepted and will be part of the 4.9 release.  The patch can be found at:  https://lkml.org/lkml/2016/8/19/474

With suspend_ops defined, the system will report the valid platform suspend states when the /sys/power/state is read.

# cat /sys/power/state
freeze mem

This example shows that both S0 (suspend to idle) and S3 (suspend to ram) are supported on this platform.  With Sudeep’s change, only freeze will show up for platforms which do not implement platform_suspend_ops.

Wake IRQ support

Once the system is placed into a sleep state, the system must receive wake events which will resume the system.  These wake events are generated from devices on the system.  It is important to make sure that device drivers utilize wake irqs and configure themselves to generate wake events upon receiving wake irqs.  If wake devices are not identified properly, the system will take the interrupt and then go back to sleep and will not resume.

Once devices implement proper wake API usage, they can be used to generate wake events.  Make sure DT files also specify wake sources properly.  An example of configuring a wakeup-source is the following (arch/arm/boot/dst/am335x-evm.dts):

     gpio_keys: volume_keys@0 {
               compatible = “gpio-keys”;
               #address-cells = <1>;
               #size-cells = <0>;
               autorepeat;

               switch@9 {
                       label = “volume-up”;
                       linux,code = <115>;
                       gpios = <&gpio0 2 GPIO_ACTIVE_LOW>;
                       wakeup-source;
               };

               switch@10 {
                       label = “volume-down”;
                       linux,code = <114>;
                       gpios = <&gpio0 3 GPIO_ACTIVE_LOW>;
                       wakeup-source;
               };
       };

As you can see, two gpio keys are defined to be wakeup-sources.  Either of these keys, when pressed, would generate a wake event during suspend.

An alternative to DT configuration is if the device driver itself configures wake support in the code using the typical wakeup facilities.

Implementation

Freeze function

Systems should define a enter_freeze() function in their cpuidle driver if they want to take full advantage of suspend to idle.  The enter_freeze() function uses a slightly different function prototype than the enter() function.  As such, you can’t just specify the enter() for both enter and enter_freeze.  At a minimum, it will directly call the enter() function.  If no enter_freeze() is specified, the suspend will occur, but the extra things that would have occurred if enter_freeze() was present, like tick_freeze() and stop_critical_timings(), will not occur.  This results in timer IRQs waking up the system.  This will not result in a resume, as the system will go back into suspend after handling the IRQ.

During suspend, minimal interrupts should occur (ideally none).

The picture below shows a plot of power usage vs time.  The two spikes on the graph are the suspend and the resume.  The small periodic spikes before and after the suspend are the system exiting idle to do bookkeeping operations, scheduling tasks, and handling timers.  It takes a certain period of time for the system to go back into the deeper idle state due to latency.

Power Usage Time Progression

The ftrace capture shown below displays the activity on the 4 CPUs before, during, and after the suspend/resume operation.  As you can see, during the suspend, no IPIs or IRQs are handled.  

Ftrace capture of Suspend/Resume

Idle State Support

You must determine which idle states support freeze.  During freeze, the power code will determine the deepest idle state that supports freeze.  This is done by iterating through the idle states and looking for which states have defined enter_freeze().  The cpuidle driver or SoC specific suspend code must determine which idle states should implement freeze and it must configure them by specifying the freeze function for all applicable idle states for each cpu.

As an example, the Qualcomm platform will set the enter_freeze function during the suspend init function in the platform suspend code.  This is done after the cpuidle driver is initialized so that all structures are defined and in place.

Driver support for Suspend/Resume

You may encounter buggy drivers during your first successful suspend operation.  Many drivers have not had robust testing of suspend/resume paths.  You may even find that suspend may not have much to do because pm_runtime has already done everything you would have done in the suspend.  Because the user space is frozen, the devices should already be idled and pm_runtime disabled.

Testing

Testing for suspend to idle can be done either manually, or through using something that does an auto suspend (script/process/etc), auto sleep or through something like Android where if a wakelock is not held the system continuously tried to suspend.  If done manually, the following will place the system in freeze:

/ # echo freeze > /sys/power/state
[  142.580832] PM: Syncing filesystems … done.
[  142.583977] Freezing user space processes … (elapsed 0.001 seconds) done.
[  142.591164] Double checking all user space processes after OOM killer disable… (elapsed 0.000 seconds)
[  142.600444] Freezing remaining freezable tasks … (elapsed 0.001 seconds) done.
[  142.608073] Suspending console(s) (use no_console_suspend to debug)
[  142.708787] mmc1: Reset 0x1 never completed.
[  142.710608] msm_otg 78d9000.phy: USB in low power mode
[  142.711379] PM: suspend of devices complete after 102.883 msecs
[  142.712162] PM: late suspend of devices complete after 0.773 msecs
[  142.712607] PM: noirq suspend of devices complete after 0.438 msecs
< system suspended >
….
< wake irq triggered >
[  147.700522] PM: noirq resume of devices complete after 0.216 msecs
[  147.701004] PM: early resume of devices complete after 0.353 msecs
[  147.701636] msm_otg 78d9000.phy: USB exited from low power mode
[  147.704492] PM: resume of devices complete after 3.479 msecs
[  147.835599] Restarting tasks … done.
/ #

In the above example, it should be noted that the MMC driver was responsible for 100ms of that 102.883ms.  Some device drivers will still have work to do when suspending.  This may be flushing of data out to disk or other tasks which take some time.

If the system has freeze defined, it will try to suspend the system.  If it does not have freeze capabilities, you will see the following:

/ # echo freeze > /sys/power/state
sh: write error: Invalid argument
/ #

Future Developments

There are two areas where work is currently being done on Suspend to Idle on ARM platforms.  The first area was mentioned earlier in the platform_suspend_ops prerequisite section.  The work to always allow for the freeze state was accepted and will be part of the 4.9 kernel.  The other area that is being worked on is the freeze_function support.

The freeze_function implementation is currently required if you want the best response/performance.  However, since most SoCs will use the ARM cpuidle driver, it makes sense for the ARM cpuidle driver to implement its own generic freeze_function.  And in fact, ARM is working to add this generic support.  A SoC vendor should only have to implement specialized freeze_functions if they implement their own cpuidle driver or require additional provisioning before entering their deepest freezable idle state.

“The first rule of all intelligent tinkering is to keep all the parts.”  — Aldo Leopold, quoted in Donald Wurster’s “Nature’s Economy”

Cloud-based continuous integration /klaʊd beɪst kənˈtɪnjʊəs ɪntɪˈɡreɪʃ(ə)n/

a software development practice that pretty much nobody would associate with microcontrollers …

I found myself last month representing Linaro at the Zephyr booth on the demo area floor in the Maritim Hotel in Berlin for ELC-E. I had a rather unusual microcontroller demo.

Microcontroller demos often tend to be a product prototype running an RTOS, often extended to sending data to a cloud-based application and/or to a mobile device. There were certainly some very cool microcontroller demos on the Zephyr booth at ELC. In our Zephyr demo we wanted to show how open the source code, tooling, and configuration could be, and how transparent collaboration could flow from the engineering behind the deeply embedded RTOS code.

Starting with some industry-standard open source tools which have been instrumental in enabling collaboration, e.g.

• git for source code management and collaboration
• standard c and make based build infrastructure
• qemu as a virtual test platform

we added some of Linaro’s existing and emerging contributions to the community in the form of

• Linaro’s open source test infrastructure – LAVA
• Continuous Integration tools, previously manifested in the Linux kernelci infrastructure
• 96Boards IoT Edition Cortex-M microcontroller-based development boards
• LITE team innovations in platform and bootloader support.

With these elements we demonstrated a continuous integration loop with testing on both virtual (cloud-hosted) targets and a local test farm of IoT-edition Cortex-M4 Carbon boards. The aim was to show how an open source RTOS like Zephyr could be complemented with best-practise open source development tools, to promote a transparent continuous collaboration environment.

In the demo, I made my modifications to the Zephyr project source code https://github.com/linaro/zephyr. In this case, the only development tool that I’m running locally is git. After I’ve made a change to the code in my local git instance, I do a push (I’ve added our CI infrastructure Zephyr repository as a remote).

That single git push command kicks off the entire test process that includes remotely building over 100 test applications, executing all of them on virtual Cortex-M devices (QEMU) and returning the results in real time to the developer.  

After the results from the virtual machine tests are executed, key target applications are subsequently built and deployed from the cloud to the small test farm of boards running in the booth and tested on multiple devices, i.e. testing on real hardware, in real-time.

Incidentally, the entire build and test infrastructure was built with scalability as as the driving factor using containers and orchestration. Even the local test farm in the booth was driven by a laptop running a container instance of the test dispatcher.

For more information on Linaro LITE please check out:

www.linaro.org/groups/lite/

Linaro and Zephyr

If you know Linaro, you’ll most likely know us as leading open source collaboration in the ARM ecosystem, and that our initial formation in 2010 saw us tackling fragmentation in the ARM Linux kernel and being the reference point for gcc toolchain support for ARM. Since then we’ve evolved to be a significant force in open source with more than 30 member companies. As well as kernel and toolchain, work includes software stacks and tools in mobile, networking, servers, the digital home and IoT.

Since I was presenting a demo that was based on Zephyr, which is a real-time operating system (RTOS), rather than on Linaro’s home turf of the Linux kernel, it’s worth explaining that earlier this year, Linaro launched its IoT and Embedded (LITE) Segment Group with a mission in the IoT and embedded space which included  

• reducing fragmentation in operating systems, middleware and cloud connectivity solutions
• providing end-to-end open source reference software for more secure connected products
• via open source solutions – enabling faster time to market, improved security and lower maintenance costs for connected products.

As a result of the decision to create this group, Linaro joined the Linux Foundation Zephyr Project as a Platinum member. The LITE group within Linaro will use Zephyr as a neutral industry RTOS platform as a place to land its collaborative engineering output.

For more information on Linaro please see www.linaro.org, and for the Zephyr project, go to www.zephyrproject.org/

More photos at: https://drive.google.com/open?id=0B1ntKia_78FZRWhNMjRzcUpTWHM

It is a well-known fact that it has been hard to get started with TEE development for a couple of reasons. For example, it has been hard to get access to the software because in the past TEE software has typically been proprietary and therefore kept within the company or under a non-disclosure agreement. On the hardware side it hasn’t been much better, and even today it is still hard to find hardware readily available for TEE development, at least if you intend to make a completely secure product. So wouldn’t it be great if we could emulate it all on a local desktop? The question is whether you actually need hardware for TEE development. As it turns out, QEMU, the machine emulator that can emulate a multitude of CPUs, officially received TrustZone support at the beginning of this year and QEMU currently supports TrustZone on both ARMv7-A and ARMv8-A architecture. But just the support in QEMU isn’t enough: you will still need the software for the TEE.

A couple of years ago Linaro, together with STMicroelectronics, teamed up and reworked STMicroelectronics proprietary TEE. That work resulted in an Open Source TEE called OP-TEE. That project has been hosted on GitHub (https://github.com/OP-TEE) since the summer of 2014. Initially it came with board support for devices coming from STMicroelectronics. But since then members of Linaro and other companies have started to use OP-TEE and today there are roughly twenty different platforms officially supported by OP-TEE. Quite early into the development of OP-TEE, Linaro added support for running OP-TEE on QEMU (ARMv7-A). Back then there were no TrustZone patches in upstream QEMU. Because of that, we were running on a fork of QEMU for quite a while. The fork contained a set of TrustZone patches that later on went into the official QEMU tree. A major reason why we did the port at an early stage was simply due to the fact that it was hard to obtain hardware back then. We could use the board from STMicroelectronics but, although it was a good development board, it had a form factor that made it hard to bring it with you, and was (and still is) hardware that is not publicly available. Today things look much better. People can choose between a variety of devices, like HiKey from Hisilicon, Raspberry Pi 3 and some devices from Freescale and TI and Xilinx. Still, with the ability to use real hardware, we haven’t let QEMU go away.

So why on earth would you want to write code to be used in a secure environment in something that isn’t secure? As it turns out, it is very convenient to use QEMU for quite a lot of the work we are doing and the turnaround time is kept to a minimum. No cables to plug and pull, no memory cards to update, no mmc to flash and, as a bonus, the GDB debugger works without the need for any modifications. All in all, you have all the tools you need running on a single computer and it doesn’t cost you anything! So how does it compare with running the code on real hardware? What is the main difference and what about the interfaces? Let’s first consider the OP-TEE boot. The ARMv8-A QEMU setup reminds more of a true boot scenario compared to ARMv7-A on QEMU. But in both cases, when running QEMU with OP-TEE, secure boot is not enabled. Having said that, there is nothing really preventing you from implementing a chain of trust, although since the boot flow is a bit different compared to other devices it doesn’t make much sense to spend time on implementing chain of trust, since it will most likely be something that won’t be used in other setups.

When the system is up and running it is another situation. There the system behaves more or less in the same way as running on real hardware. The biggest limitation using QEMU is that some peripherals might not be emulated and some low level functionality might not be fully supported (CPU caches etc). But for developing and debugging the core of the TEE (kernel mode in secure world) you can do almost everything and, most of the time, the changes you have made will work when you have compiled for another platform and try it out on real hardware.

What about the Trusted Applications? If you have some prior knowledge about the GlobalPlatform Internal Core API specification, you know that this specification specifies how to deal with cryptographic operations, secure storage, secure time and how to work the arithmetical (big number) operations. So the question is: do the Trusted Applications need some special treatment to work with QEMU? The short answer is no. Everything but secure time works just fine, and secure time is heavily platform dependent regardless if you’re running QEMU or not. The cryptographic operations run using a software implementation and secure storage uses whatever root file system you have decided to use. We typically use a initramfs based file system. Everything is transparent to the one writing the Trusted Application. The important thing is that Trusted Applications being developed for use with OP-TEE should be written so they use and follow the rules in the API specified by the GlobalPlatform TEE Internal Core API specification. By doing so, there is no need to make any changes to the source code at all when building for different platforms and devices. In fact, by following a standard strictly, I believe it is possible to take the source code as it is and compile it using another TEE vendors’ SDKs and development kits without having to make many changes at all. A missing piece, but a great step in the future, would be to create Trusted Applications that are binary compatible so you wouldn’t have to recompile for different platforms or even when running on a different TEE solution. Unfortunately, I think we are a bit far away from that now.

As mentioned, GDB just works. QEMU provides a GDB stub (the -s parameter), which means that you not only have the ability to develop and test the solution, but that you can also debug the entire solution, i.e., Linux kernel, TEE core and the Trusted Applications. By default the debugger is text based, but there are some quite decent graphical interfaces available if someone prefers using that instead. From a debugger point of view everything works as expected: you can set breakpoints, examine variables, memory etc. This is a very powerful tool and can save a lot of time and headaches both when trying to find bugs and when studying the code. Again, this doesn’t cost you anything.

The 2016 developer workshop, hosted by GlobalPlatform in Santa Clara, covered everything discussed above. The workshop covered how to write code and debug the TEE core itself and how to write, deploy and test a Trusted Application using QEMU and OP-TEE, and attendees learned how to work with OP-TEE using QEMU. The takeaway was that using a local setup as described here is a good way for people new to TEE development to get started, and to make experienced users’ lives a bit easier when they are working with real products, by simplifying the setup and minimizing the turnaround time. Most likely the majority of their development could be done by using QEMU and then finalizing the remaining bits and pieces on the target hardware.

For users who did not attend the workshop, but still would like to try this, we recommend that you head over to https://github.com/OP-TEE/optee_os and read through the README.md file (prerequisites in section 4 and QEMU in section 5 should be sufficient to get a working setup by running a handful of command in a Linux shell).

This blog was originally posted the TeeSeminar.org site:  http://www.teeseminar.org/media_center_blog_jbech.asp

“What we wish, that we readily believe.”  — Demosthenes