Difference between revisions of "LivePatch"

From Xen
(Hypervisor)
(Tech Preview -> SUPPORTED)
 
(82 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
= Design =
 
= Design =
   
<s>[http://lists.xen.org/archives/html/xen-devel/2015-07/msg04951.html xSplice RFC v3.1 design]</s>
+
Xen 4.7 and later implements v1 of the [http://xenbits.xen.org/docs/unstable/misc/livepatch.html Live Patch Design].
[http://lists.xen.org/archives/html/xen-devel/2015-09/msg02116.html Xsplice v1 design]
 
   
  +
There are three parts to utilize this technology:
<s>On August 18th @11:30 - 12:20 in Seattle will nail down in person some of the questions
 
  +
* Hypervisor implementing the LIVEPATCH_SYSCTL hypercall
raised on xen-devel mailing list.</s>Done - code has been updated to include resolution on that.
 
  +
* Tool to upload/apply/revert the payloads: xen-livepatch
  +
* Tool to generate the payloads.
   
  +
The first two are part of the Xen Project source:
The git code for hypervisor/toolstack currently is at:
 
git://xenbits.xen.org/people/konradwilk/xen.git #xsplice.v1.1
 
   
  +
git://xenbits.xen.org/xen.git
The generation of payload is at:
 
https://github.com/rosslagerwall/xsplice-build
 
   
  +
while the last one is in a seperate repository:
= IRC =
 
   
  +
git://xenbits.xen.org/livepatch-build-tools.git
We will have monthly IRC meetings to discuss code/issues/etc.
 
   
They are on the fourth Monday of every month. At 10:00AM EST on #xsplice channel on irc.freenode.net
 
   
  +
== How to enable it ==
= Toolstack =
 
   
  +
The code for hypervisor is enabled by default (for x86). When compiling the hypervisor one must
Things that need to be done in there:
 
  +
change the config to build it.
* <s>xen-xsplice better smarts. Right now you need to do 'upload','check' and 'apply'. Three operations which could be folded in one - since the system admin probably want to have all of those done at once.</s> The 'all' option does that.
 
* Instead of using 'xen-xsplice' we should use 'xl'. That above mentioned logic could be part of 'xl' while 'xen-xsplice' is an accessory function for expert users?
 
   
  +
It can be done by editing <i>xen/.config</i> to have <b>CONFIG_LIVEPATCH=y</b> or using:
= Build =
 
   
  +
$ git clone git://xenbits.xen.org/xen.git
* <s>Compile Xen hypervisor with -ffunction-sections.</s> (Ross)
 
Oct 12 2015: "
+
$ cd xen/xen
  +
$ make menuconfig
I've created a _prototype_ tool for this based on kpatch's tooling. It's
 
currently living at https://github.com/rosslagerwall/xsplice-build
 
With no source patch modifications, live patches can be built for every XSA
 
that applies to x86 back to XSA-90 except for XSA-97, XSA-111, XSA-112, and
 
XSA-114 (83% success rate). It gives plausible output for each generated patch
 
although I obviously can't verify the live patches yet.
 
It doesn't really follow the design in the above document; IMO the payload
 
design is unnecessarily complicated. At this point I'd rather just get a
 
complete working prototype, and we can figure out the specifics and finalize
 
the design later."
 
   
  +
And selecting Common Features|Live Patch live patching support.
= Hypervisor =
 
   
  +
It is marked as <b>SUPPORTED</b> for x86. For ARM 32&64 it is <b>TECH PREVIEW</b>.
We need to expand the hypervisor to include support for:
 
* <s>build_id support (done!)</s>
 
Oct-12-2015: Konrad posted v1 patchset. Needs rework.
 
* Need 'dl_sym' functionality in hypervisor to parse ELF payload. Do dynamic linking. (Ross)
 
* Lookup and insertion in the symbol table. Compute the proper offset and virtual address based on symbol name (<do_domctl>+0xf) (Ross)
 
* Need to either remove duplicate names from the symbol table or have the hypervisor implement proper ELF parsing - integrate the elf sections and parse that during runtime.
 
* Insertion of exceptions in the exception table. Also support resorting it. The hot-patch may have new exceptions that need to be dealt with.
 
* Code to verify signature of the ELF payload.
 
* Checking the build-id against the hypervisor.
 
* Code to patch and revert the hot-patch (for simplicity we could boot Xen with 'cpus=1' so we don't have to worry about proper code to trigger patching)
 
** Patching requires saving the initial state. That needs to be saved for revert.
 
** Patching needs allocation of memory, putting in the code at memory.
 
** Doing the verification - checking if any code that uses the to-be-patched-function is in use. Save what the virtual address for the functions is - and verify the stack when patching (this incidentally is what .xsplice_section structures can have).
 
** Deal with proper page table types for the modification (ro for .data, x for .text, etc).
 
** Code to revert.
 
* Code to activate patching mechanism. On VMEXIT routine right before checking the do_softirq. Ditto for PV paths. Call the top patching mechanism - which can IPI other CPUs (or wait until all CPUs have come to the same point).
 
* The code for doing the patching - call from __do_softirq - can also be used for [http://wiki.xenproject.org/wiki/XenParavirtOps/microcode_update runtime microcode loading].
 
* Add functionality to execute ELF code.
 
   
  +
Once you have it built and booted, you can verify that Live Patch is built in via:
= ELF payload file =
 
   
  +
#xl dmesg | grep livepatch
The spec details how the structures should look. We need to write tools
 
  +
(XEN) livepatch: : build-id: 34c63348019acbe8bab69808a6f8ad7618fa7c12
to create this ELF file based on a header (or C code). This could be a simple as
 
linker script magic. (Konrad wrote linker script, but the tool is primitive and a hack)
 
   
  +
(which you will only see if you build the hypervisor with debug=y)
= Create regression hot-patches =
 
  +
or alternatively via:
  +
# xl info | grep build_id
  +
build_id : 34c63348019acbe8bab69808a6f8ad7618fa7c12
   
  +
The value: 34c63348019acbe8bab69808a6f8ad7618fa7c12 is different on every build.
Need to create an .xsplice ELF payload which changes xen_extra .data structure. This way we can verify the applying and reverting of a hot-patch - and check for expected output via 'xl info' and string comparison. To make this work:
 
  +
(This build-id is used to keep track of payloads dependencies and make sure they are built
* Write script (or C code?) which generates an ELF payload with the required changes to patch 'xen_extra'.
 
  +
against the correct hypervisor built).
  +
  +
The tool to patch/load/etc is called <b>xen-livepatch</b> and is also part of the Xen source tree.
  +
It is used to load/apply/revert/unload the payloads.
  +
  +
== ELF payload file ==
  +
  +
The [http://xenbits.xen.org/docs/unstable/misc/livepatch.html design] details how the structures look, see <b>Design of payload format</b> for details.
  +
  +
The external tool:
  +
  +
git://xenbits.xen.org/livepatch-build-tools.git
  +
  +
can generate the payloads against the hypervisor.
   
 
= Build hot-patches =
 
= Build hot-patches =
  +
* Compile Xen hypervisor where the output (.o) ends up in (.o.xsplice) to make it easier to identify changes. Could be a secondary build ('make xsplice')
 
  +
To build hot-patches aka payloads aka binary fixes against the hypervisor, you need
* Write in scripts/ code to compare the .o and .o.xsplice files using objcopy, diff, etc to figure out if they changed. If they do, copy the 'new' and 'old' object file as '.o.xsplice.baseline' and '.o.xsplice.new'.
 
  +
two things:
* Write tool to look for '.o.xsplice.baseline' and '.o.xsplice.new' - figure out the ELF relocations, what symbols it affects, etc. Take a hard look at objmanip.c in ksplice git tree (git://repo.or.cz/ksplice.git).
 
  +
* To start with this doing C code and injecting new code can be used as a prototype to develop.
 
  +
* Xen hypervisor which supports Live Patch hypercalls.
* Look at the 30 last XSAa and use that as a measuring stick of how well the hot-patching build works.
 
  +
* Tool to build binary fixes against hypervisor.
  +
  +
The tool is available at at:
  +
git://xenbits.xen.org/livepatch-build-tools.git
  +
  +
Or alternatively the three examples that are part of the Xen code base.
  +
  +
== How to build built-in examples ==
  +
  +
The Xen Project also includes three (four in Xen 4.8) regression test-cases that can be built
  +
against the hypervisor which modify the <i>xen_extra_version</i>. This provides
  +
an simple way to verify the changes as:
  +
  +
# xl info | grep extra_version
  +
  +
will verify that the patch has taken place.
  +
  +
There are three regression/test-cases:
  +
  +
* xen_hello_world: changes xen_extra_version() to return "Hello World".
  +
* xen_bye_world: over-writes the xen_hello_world patch so it returns "Bye World."
  +
* xen_replace_world: Used to replace the two other payloads and make xen_extra_version() print "Hello Again World".
  +
* xen_nop: Used to NOP xen_minor_version - which results in it reporting Xen 4.4
  +
  +
To build them:
  +
  +
<syntaxhighlight lang='sh'>
  +
$ cd xen
  +
$ make -C xen tests
  +
...
  +
$ find . -name *.livepatch
  +
./xen/arch/x86/test/xen_replace_world.livepatch
  +
./xen/arch/x86/test/xen_hello_world.livepatch
  +
./xen/arch/x86/test/xen_bye_world.livepatch
  +
./dist/install/usr/lib/debug/xen_replace_world.livepatch
  +
./dist/install/usr/lib/debug/xen_hello_world.livepatch
  +
./dist/install/usr/lib/debug/xen_bye_world.livepatch
  +
</syntaxhighlight>
  +
  +
<b>NOTE</b>In Xen 4.8 and later versions they are in xen/test/livepatch directory.
  +
  +
<b>NOTE</b> Since they are built on the same source and binary as the hypervisor that is running - it automatically extracts the build-id and embeds the build-id.
  +
  +
  +
You can use an simple Perl script located at [http://xenbits.xen.org/gitweb/?p=xentesttools/bootstrap.git;a=blob_plain;f=root_image/debugspace/livepatch_test.pl;hb=HEAD; livepatch_test.pl]
  +
to perform a variety of test-cases to see whether the livepatch functionlity works.
  +
  +
The idea is to do (on host with Xen hypervisor that has Live Patch built-in):
  +
<syntaxhighlight lang='sh'>
  +
$ # copy them the host.
  +
$ cd /usr/lib/debug
  +
$ ls
  +
xen-syms-4.7-unstable xen_bye_world.livepatch xen_hello_world.livepatch
  +
xen_replace_world.livepatch
  +
$ sudo /root/livepatch_test.pl
  +
</syntaxhighlight>
  +
  +
The tool should return an error code zero.
  +
  +
<b>OLD INSTRUCTIONS</b>
  +
  +
The idea is to do (on host with Xen hypervisor that has Live Patch built-in):
  +
  +
<syntaxhighlight lang='sh'>
  +
$ # copy them the host.
  +
# cd /usr/lib/debug
  +
# ls
  +
xen-syms-4.7-unstable xen_bye_world.livepatch xen_hello_world.livepatch
  +
xen_replace_world.livepatch
  +
# xen-livepatch load xen_hello_world.livepatch
  +
Uploading xen_hello_world.livepatch (12336 bytes)
  +
Performing apply:. completed
  +
# xl info | grep extra
  +
xen_extra : Hello World
  +
# xen-livepatch load xen_bye_world.livepatch
  +
Uploading xen_bye_world.livepatch (9244 bytes)
  +
Performing apply:. completed
  +
# xl info | grep extra
  +
xen_extra : Bye World!
  +
# xen-livepatch upload replace_bye_and_hello xen_replace_world.livepatch
  +
Uploading xen_replace_world.livepatch (9328 bytes)
  +
# xen-livepatch list
  +
ID | status
  +
----------------------------------------+------------
  +
xen_hello_world | APPLIED
  +
xen_bye_world | APPLIED
  +
replace_bye_and_hello | CHECKED
  +
# xen-livepatch replace replace_bye_and_hello
  +
Performing replace:. completed
  +
# xl info | grep extra
  +
xen_extra : Hello Again World!
  +
# xen-livepatch list
  +
ID | status
  +
----------------------------------------+------------
  +
xen_hello_world | CHECKED
  +
xen_bye_world | CHECKED
  +
replace_bye_and_hello | APPLIED
  +
</syntaxhighlight>
  +
  +
== livepatch-build-tools ==
  +
  +
The workflow of the tool is to:
  +
* Build a Xen hypervisor without the patch (alternatively one can just point it to <b>xen-syms</b> which is preferred.
  +
* Build a Xen hypervisor with a patch.
  +
* Compare the two above.
  +
* Create an payload with the difference.
  +
  +
The tool is <b>not</b> capable of:
  +
* Generating payloads only with .data changes (as in only for variables),
  +
* Inline patching - it only patches the whole function,
  +
* Cannot NOP out functions.
  +
  +
This explanation was lifted from [http://lists.xenproject.org/archives/html/xen-devel/2015-10/msg02595.html this link] and modified a bit.
  +
  +
This transcript should demonstrate how to use the tool. We will create three patches:
  +
* a) First depends on the hypervisor
  +
* b) Second depends on the first one being applied (cumulative)
  +
* c) Third one replaces the first a) and second one b).
  +
  +
<b>NOTE</b>The a) and c) payloads need to embedded an build-id of the hypervisor it will be loaded against.
  +
That means one <b>SHOULD NOT</b> recompile the source tree of the Xen hypervisor that is running.
  +
Hence the workload here is done on a copy of the source. The b) depends on the build-id of the a) payload.
  +
  +
This assumes that the <b>xen-syms</b> built originally is in <i>~/xen-orig/xen/xen-syms</i>
  +
We use /tmp to store a copy of the source. The version of the source code
  +
<b>MUST</b> match the version that <i>/xen-orig/xen/xen-syms</i> was built.
  +
One can easily verify that by (on target system):
  +
  +
  +
# xl info | grep xen_changeset
  +
$ Fri Apr 15 10:35:21 2016 -0400 git:7552904
  +
  +
The build-id in this example is <i>34c63348019acbe8bab69808a6f8ad7618fa7c12</i> but it will
  +
be different on each build. One can extract this by:
  +
  +
<syntaxhighlight lang='sh'>
  +
# xl info | grep build_id
  +
build_id : 34c63348019acbe8bab69808a6f8ad7618fa7c12
  +
  +
  +
$ cd /tmp/
  +
$ git clone git://xenbits.xen.org/xen.git
  +
$ # Or alternatively one can do git clone ~/xen-orig
  +
$ cd xen
  +
$ git checkout 7552904
  +
  +
$ # Copy .config from the built hypervisor.
  +
$ cp ~/xen-orig/xen/.config /tmp/config
  +
$ Make sure LIVEPATCH is enabled
  +
$ cat /tmp/config | grep LIVEPATCH
  +
CONFIG_LIVEPATCH=y
  +
  +
$ # Write a patch
  +
$ mkdir ~/work
  +
$ git diff > ~/work/test1.patch
  +
$ git reset --hard
  +
$ # Write another patch
  +
$ git diff > ~/work/test2.patch
  +
$ git reset --hard
  +
$ # Write another patch
  +
$ git diff > ~/work/test3.patch
  +
$ git reset --hard
  +
  +
$ cd /tmp
  +
$ git clone git://xenbits.xen.org/livepatch-build-tools.git
  +
$ cd livepatch-build-tools
  +
$ make
  +
$ ./livepatch-build -s /tmp/xen -c /tmp/config -p ~/work/test1.patch -o out1 -d --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12
  +
</syntaxhighlight>
  +
  +
<b>NOTE</b> Don't blindly plug in the build-id value! Use the one your hypervisor is built with.
  +
<b>NOTE</b> If you are using a pre-release version also use <b--xen-debug</b>.
  +
  +
Or alternatively make it automatic if you do this often:
  +
  +
--depends $(readelf -Wn ~/xen-orig/xen/xen-syms | tail -1 | sed 's/.*Build ID: //')
  +
  +
  +
The first Live Patch payload depends on the hypervisor build-id, while the second
  +
one (test2) will depend on the first one (test1).
  +
  +
Extract the build-id of the first payload:
  +
  +
$ readelf -Wn out1/test1.livepatch| tail -1 | sed 's/.*Build ID: //'
  +
dbbd14b6aa110b9065e2667d42ef4291eed6b821
  +
  +
<b>NOTE</b> The build-id varies per build. This value is just an example - yours will differ.
  +
  +
<b>NOTE</b> We use <b>--xen-syms</b> - for cumulative patches (one payload on top of another) it is imperative that we have the original. This is because there's no way that the hot patch can prelink against a function introduced by a new patch because it is placed in a random area of memory.
  +
  +
For this, you need to pass the xen-syms that is actually being used, so all the addresses when resolving symbols are precisely correct:
  +
  +
<syntaxhighlight lang='sh'>
  +
$ ./livepatch-build -s /tmp/xen -p ~/work/test2.patch -o out2 -c /tmp/config -d --xen-syms ~/xen-orig/xen/xen-syms --depends dbbd14b6aa110b9065e2667d42ef4291eed6b821
  +
</syntaxhighlight>
  +
  +
And the last one - build against the hypervisor:
  +
  +
<syntaxhighlight lang='sh'>
  +
$ ./livepatch-build -s /tmp/xen -p ~/work/test3.patch -o out3 -c /tmp/config --debug --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12
  +
  +
$ # copy out*/test*.livepatch onto the host (or if locally skip this step)
  +
  +
  +
On the host:
  +
# xen-livepatch upload test1 test1.livepatch
  +
# xen-livepatch upload test2 test2.livepatch
  +
# xen-livepatch upload test3 test3.livepatch
  +
# xen-livepatch list
  +
ID | status
  +
----------------------------------------+------------
  +
test1 | CHECKED
  +
test2 | CHECKED
  +
test3 | CHECKED
  +
# # verify the build-ids match
  +
# xl debug-keys x
  +
# xl dmesg | tail -15
  +
(XEN) name=test1 state=CHECKED(1) ffff82d080a09000 (.data=ffff82d080a0a000, .rodata=ffff82d080a0a000) using 2 pages.
  +
(XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a09000 (1956)
  +
(XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a097c8 (1879)
  +
(XEN) build-id=dbbd14b6aa110b9065e2667d42ef4291eed6b821
  +
(XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12
  +
(XEN) name=test2 state=CHECKED(1) ffff82d080a0c000 (.data=ffff82d080a0d000, .rodata=ffff82d080a0d000) using 2 pages.
  +
(XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0c000 (1887)
  +
(XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a0c783 (1964)
  +
(XEN) build-id=50364b9dbdb57e15f57bac237fce7fdfc37a94bd
  +
(XEN) depend-on=dbbd14b6aa110b9065e2667d42ef4291eed6b821
  +
(XEN) name=test3 state=CHECKED(1) ffff82d080a0f000 (.data=ffff82d080a10000, .rodata=ffff82d080a10000) using 2 pages.
  +
(XEN) do_xen_version patch ffff82d080115069(1930) with ffff82d080a0f000 (1958)
  +
(XEN) compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0f7ca (1881)
  +
(XEN) build-id=0f2d152953babf40b6635e2c80adcb3ced2b04d9
  +
(XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12
  +
  +
# xen-livepatch apply test1
  +
# # Verify test1 is applied
  +
# xen-livepatch apply test2
  +
# # Verify test2 is also applied
  +
# xen-livepatch list
  +
ID | status
  +
----------------------------------------+------------
  +
test1 | APPLIED
  +
test2 | APPLIED
  +
test3 | CHECKED
  +
  +
# xen-livepatch replace test3
  +
# # Verify test3 is applied and test1 and test2 are not
  +
# xen-livepatch revert test3
  +
# # Verify test3 is not applied
  +
  +
# xen-livepatch unload test1
  +
# xen-livepatch unload test2
  +
# xen-livepatch unload test3
  +
</syntaxhighlight>
  +
  +
= TODOs =
  +
  +
v1 of patchset is in the Xen code base. We still have a list of TODOs and track it here.
  +
  +
== OSSTest ==
  +
  +
<s>Make OSSTest do regression tests.<s> Xen 4.9
  +
  +
== Toolstack ==
  +
  +
Things that need to be done in there:
  +
* <s>xen-livepatch better smarts. Right now you need to do 'upload','check' and 'apply'. Three operations which could be folded in one - since the system admin probably want to have all of those done at once.</s> The 'all' option does that.
  +
* Instead of using 'xen-livepatch' we should use 'xl'. That above mentioned logic could be part of 'xl' while 'xen-livepatch' is an accessory function for expert users?
  +
* If hypervisor is compiled without Live Patch the tool should print simply '"livepatch support not available in Xen'
  +
* If the build-id is wrong and one tries to apply the payload the message of 'hello failed with 1(Operation not permitted)' is lacking. We should have more details (and maybe a different error?). Or perhaps check the build-d as as being upload (however we may load them out of order). Perhaps have the toolstack check the build-id? How to do that against loaded patches (have them locally saved?)
  +
* Read build-id of the payloads files and display them - as well the dependencies. Perhaps even add a simple dependency checker.
  +
  +
== livepatch-build--tools ==
  +
  +
* Provide better error-code - saying what is unsupported if the user tries to do it.
  +
* Able to generate payloads that NOP functions.
  +
* Able to generate payloads only against .data sections.
  +
* Create payloads for ARM 32
  +
* Create payloads for ARM 64
  +
  +
== Hypervisor ==
  +
  +
Currently the implementation has many TODOs implemented.
  +
  +
However we need to expand the hypervisor to include support for:
  +
* <s>build_id support</s> Done! (4.7)
  +
* <s>The check for --build-id needs to deal with localized binutils.</s> (4.7)
  +
* <s>Make it work with .rodata changes only, or just inline changes.</s> (4.8)
  +
* <s>Need 'dl_sym' functionality in hypervisor to parse ELF payload. Do dynamic linking. (Ross)</s> (4.7)
  +
* <s>Lookup and insertion in the symbol table. Compute the proper offset and virtual address based on symbol name (<do_domctl>) (Ross)</s> (4.7)
  +
* <s>Need to either remove duplicate names from the symbol table or have the hypervisor implement proper ELF parsing - integrate the elf sections and parse that during runtime.</s> (4.7)
  +
* <s>Insertion of exceptions in the exception table. Also support resorting it. The hot-patch may have new exceptions that need to be dealt with.</s> (4.7)
  +
* <s>Checking the build-id against the hypervisor.<s/> (4.7)
  +
* <s>Code to patch and revert the hot-patch (for simplicity we could boot Xen with 'cpus=1' so we don't have to worry about proper code to trigger patching)</s> (4.7)
  +
** <s>Support NMI? (or something similar?)</s> (4.7)
  +
** <s>Patching requires saving the initial state. That needs to be saved for revert.</s> (4.7)
  +
** <s>Patching needs allocation of memory, putting in the code at memory.</s> (4.7)
  +
** <s>Deal with proper page table types for the modification (ro for .data, x for .text, etc).</s> (4.7)
  +
** <s>Code to revert.</s> (4.7)
  +
* <s>Code to activate patching mechanism. On VMEXIT routine right before checking the do_softirq. Ditto for PV paths. Call the top patching mechanism - which can IPI other CPUs (or wait until all CPUs have come to the same point).</s> (4.7)
  +
* <s>Add functionality to execute ELF code.</s> (4.8)
  +
* <s>Patch with NOPs if .livepatch.funcs->new_size has 0.</s> (4.8)
  +
* <s>Seperate pages for .text; .rodata; .bss and .data</s> (4.8)
  +
* <s>Make it work under ARM32:</s>Done! (4.8)
  +
** <s>Support ARM32 ELF relocations</s>
  +
** <s>Support ARM32 virtual addresses</s>
  +
** <s>Support a similar to CPU flushing semantics as x86</s>
  +
** <s>Support NMI? (or something similar?)</s>
  +
* <s>Make it work under ARM64:</s> (4.8)
  +
** <s>Support ARM64 ELF relocations</s>
  +
** <s>Support ARM64 virtual addresses</s>
  +
** <s>Support a similar to CPU flushing semantics as x86</s>
  +
* A better mechanism to "mask" NMIs during patching. The existing mechanism looses NMI if they have been sent and we don't have a mechanism to replay them. Note that this is also fixes alternative section patching. Could (like Linux) annotate handlers don't get patched.
  +
* Code to verify signature of the ELF payload.
  +
** Doing the verification - checking if any code that uses the to-be-patched-function is in use. Save what the virtual address for the functions is - and verify the stack when patching (this incidentally is what .livepatch_section structures can have).
  +
* tboot - will need to remeasure hypervisor when adding new code
  +
* SecureBoot - not sure how it works with Live Patch. Needs to be tested. Xen should verify payload - and in turn use SecureBoot services to verify?
  +
* Make .livepatch.funcs be an SHF_MERGE type with .sh_entsize having the size of the struct.
  +
* Make the usage of tools/symbols use common list model (similar to how alternative code is built)
  +
* When figuring out the new_addr and the name is <symbol>+<offset> - compute that.
  +
* The code for doing the patching - call from __do_softirq - can also be used for [http://wiki.xenproject.org/wiki/XenParavirtOps/microcode_update runtime microcode loading].
  +
* Make XENPF_get_symbol also include Live Patch symbols.
  +
  +
[[Category:Security]]
  +
[[Category:LivePatch]]
  +
[[Category:Xen 4.7]]

Latest revision as of 13:34, 5 October 2017

Design

Xen 4.7 and later implements v1 of the Live Patch Design.

There are three parts to utilize this technology:

  • Hypervisor implementing the LIVEPATCH_SYSCTL hypercall
  • Tool to upload/apply/revert the payloads: xen-livepatch
  • Tool to generate the payloads.

The first two are part of the Xen Project source:

git://xenbits.xen.org/xen.git

while the last one is in a seperate repository:

git://xenbits.xen.org/livepatch-build-tools.git


How to enable it

The code for hypervisor is enabled by default (for x86). When compiling the hypervisor one must change the config to build it.

It can be done by editing xen/.config to have CONFIG_LIVEPATCH=y or using:

 $ git clone git://xenbits.xen.org/xen.git
 $ cd xen/xen
 $ make menuconfig

And selecting Common Features|Live Patch live patching support.

It is marked as SUPPORTED for x86. For ARM 32&64 it is TECH PREVIEW.

Once you have it built and booted, you can verify that Live Patch is built in via:

 #xl dmesg | grep livepatch
 (XEN) livepatch: : build-id: 34c63348019acbe8bab69808a6f8ad7618fa7c12

(which you will only see if you build the hypervisor with debug=y) or alternatively via:

 # xl info | grep build_id
 build_id               : 34c63348019acbe8bab69808a6f8ad7618fa7c12

The value: 34c63348019acbe8bab69808a6f8ad7618fa7c12 is different on every build. (This build-id is used to keep track of payloads dependencies and make sure they are built against the correct hypervisor built).

The tool to patch/load/etc is called xen-livepatch and is also part of the Xen source tree. It is used to load/apply/revert/unload the payloads.

ELF payload file

The design details how the structures look, see Design of payload format for details.

The external tool:

git://xenbits.xen.org/livepatch-build-tools.git

can generate the payloads against the hypervisor.

Build hot-patches

To build hot-patches aka payloads aka binary fixes against the hypervisor, you need two things:

  • Xen hypervisor which supports Live Patch hypercalls.
  • Tool to build binary fixes against hypervisor.

The tool is available at at:

 git://xenbits.xen.org/livepatch-build-tools.git 

Or alternatively the three examples that are part of the Xen code base.

How to build built-in examples

The Xen Project also includes three (four in Xen 4.8) regression test-cases that can be built against the hypervisor which modify the xen_extra_version. This provides an simple way to verify the changes as:

 # xl info | grep extra_version

will verify that the patch has taken place.

There are three regression/test-cases:

  • xen_hello_world: changes xen_extra_version() to return "Hello World".
  • xen_bye_world: over-writes the xen_hello_world patch so it returns "Bye World."
  • xen_replace_world: Used to replace the two other payloads and make xen_extra_version() print "Hello Again World".
  • xen_nop: Used to NOP xen_minor_version - which results in it reporting Xen 4.4

To build them:

  $ cd xen
  $ make -C xen tests
  ...
  $ find . -name *.livepatch
  ./xen/arch/x86/test/xen_replace_world.livepatch
  ./xen/arch/x86/test/xen_hello_world.livepatch
  ./xen/arch/x86/test/xen_bye_world.livepatch
  ./dist/install/usr/lib/debug/xen_replace_world.livepatch
  ./dist/install/usr/lib/debug/xen_hello_world.livepatch
  ./dist/install/usr/lib/debug/xen_bye_world.livepatch

NOTEIn Xen 4.8 and later versions they are in xen/test/livepatch directory.

NOTE Since they are built on the same source and binary as the hypervisor that is running - it automatically extracts the build-id and embeds the build-id.


You can use an simple Perl script located at livepatch_test.pl to perform a variety of test-cases to see whether the livepatch functionlity works.

The idea is to do (on host with Xen hypervisor that has Live Patch built-in):

  $ # copy them the host.
  $ cd /usr/lib/debug
  $ ls
  xen-syms-4.7-unstable  xen_bye_world.livepatch  xen_hello_world.livepatch  
  xen_replace_world.livepatch  
  $ sudo /root/livepatch_test.pl

The tool should return an error code zero.

OLD INSTRUCTIONS

The idea is to do (on host with Xen hypervisor that has Live Patch built-in):

  $ # copy them the host.
  # cd /usr/lib/debug
  # ls
  xen-syms-4.7-unstable  xen_bye_world.livepatch  xen_hello_world.livepatch  
  xen_replace_world.livepatch  
  # xen-livepatch load xen_hello_world.livepatch 
  Uploading xen_hello_world.livepatch (12336 bytes)
  Performing apply:. completed
  # xl info | grep extra
  xen_extra              : Hello World
  # xen-livepatch load xen_bye_world.livepatch 
  Uploading xen_bye_world.livepatch (9244 bytes)
  Performing apply:. completed
  # xl info | grep extra
  xen_extra              : Bye World!
  # xen-livepatch upload replace_bye_and_hello xen_replace_world.livepatch 
  Uploading xen_replace_world.livepatch (9328 bytes)
  # xen-livepatch list
   ID                                     | status
  ----------------------------------------+------------
  xen_hello_world                         | APPLIED
  xen_bye_world                           | APPLIED
  replace_bye_and_hello                   | CHECKED
  # xen-livepatch replace replace_bye_and_hello
  Performing replace:. completed
  # xl info | grep extra
  xen_extra              : Hello Again World!
  # xen-livepatch list
   ID                                     | status
  ----------------------------------------+------------
  xen_hello_world                         | CHECKED
  xen_bye_world                           | CHECKED
  replace_bye_and_hello                   | APPLIED

livepatch-build-tools

The workflow of the tool is to:

  • Build a Xen hypervisor without the patch (alternatively one can just point it to xen-syms which is preferred.
  • Build a Xen hypervisor with a patch.
  • Compare the two above.
  • Create an payload with the difference.

The tool is not capable of:

  • Generating payloads only with .data changes (as in only for variables),
  • Inline patching - it only patches the whole function,
  • Cannot NOP out functions.

This explanation was lifted from this link and modified a bit.

This transcript should demonstrate how to use the tool. We will create three patches:

  • a) First depends on the hypervisor
  • b) Second depends on the first one being applied (cumulative)
  • c) Third one replaces the first a) and second one b).

NOTEThe a) and c) payloads need to embedded an build-id of the hypervisor it will be loaded against. That means one SHOULD NOT recompile the source tree of the Xen hypervisor that is running. Hence the workload here is done on a copy of the source. The b) depends on the build-id of the a) payload.

This assumes that the xen-syms built originally is in ~/xen-orig/xen/xen-syms We use /tmp to store a copy of the source. The version of the source code MUST match the version that /xen-orig/xen/xen-syms was built. One can easily verify that by (on target system):


 # xl info | grep xen_changeset
 $ Fri Apr 15 10:35:21 2016 -0400 git:7552904

The build-id in this example is 34c63348019acbe8bab69808a6f8ad7618fa7c12 but it will be different on each build. One can extract this by:

  # xl info | grep build_id
  build_id               : 34c63348019acbe8bab69808a6f8ad7618fa7c12


  $ cd /tmp/
  $ git clone git://xenbits.xen.org/xen.git
  $ # Or alternatively one can do git clone ~/xen-orig
  $ cd xen
  $ git checkout 7552904
  
  $ # Copy .config from the built hypervisor.
  $ cp ~/xen-orig/xen/.config /tmp/config
  $ Make sure LIVEPATCH is enabled   
  $ cat /tmp/config | grep LIVEPATCH
  CONFIG_LIVEPATCH=y

  $ # Write a patch
  $ mkdir ~/work
  $ git diff > ~/work/test1.patch
  $ git reset --hard
  $ # Write another patch
  $ git diff > ~/work/test2.patch
  $ git reset --hard
  $ # Write another patch
  $ git diff > ~/work/test3.patch
  $ git reset --hard

  $ cd /tmp
  $ git clone git://xenbits.xen.org/livepatch-build-tools.git
  $ cd livepatch-build-tools
  $ make
  $ ./livepatch-build  -s /tmp/xen -c /tmp/config -p ~/work/test1.patch -o out1 -d --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12

NOTE Don't blindly plug in the build-id value! Use the one your hypervisor is built with. NOTE If you are using a pre-release version also use <b--xen-debug.

Or alternatively make it automatic if you do this often:

 --depends $(readelf -Wn ~/xen-orig/xen/xen-syms |  tail -1 | sed 's/.*Build ID: //')


The first Live Patch payload depends on the hypervisor build-id, while the second one (test2) will depend on the first one (test1).

Extract the build-id of the first payload:

 $ readelf -Wn out1/test1.livepatch| tail -1 | sed 's/.*Build ID: //'
 dbbd14b6aa110b9065e2667d42ef4291eed6b821

NOTE The build-id varies per build. This value is just an example - yours will differ.

NOTE We use --xen-syms - for cumulative patches (one payload on top of another) it is imperative that we have the original. This is because there's no way that the hot patch can prelink against a function introduced by a new patch because it is placed in a random area of memory.

For this, you need to pass the xen-syms that is actually being used, so all the addresses when resolving symbols are precisely correct:

  $ ./livepatch-build -s /tmp/xen -p ~/work/test2.patch -o out2 -c /tmp/config -d --xen-syms ~/xen-orig/xen/xen-syms --depends dbbd14b6aa110b9065e2667d42ef4291eed6b821

And the last one - build against the hypervisor:

  $ ./livepatch-build -s /tmp/xen -p ~/work/test3.patch -o out3 -c /tmp/config --debug --depends 34c63348019acbe8bab69808a6f8ad7618fa7c12

  $ # copy out*/test*.livepatch onto the host (or if locally skip this step)
  
  
  On the host:
  # xen-livepatch upload test1 test1.livepatch
  # xen-livepatch upload test2 test2.livepatch
  # xen-livepatch upload test3 test3.livepatch
  # xen-livepatch list
   ID                                     | status
  ----------------------------------------+------------
  test1                                   | CHECKED
  test2                                   | CHECKED
  test3                                   | CHECKED
  # # verify the build-ids match
  # xl debug-keys x
  # xl dmesg | tail -15
  (XEN)  name=test1 state=CHECKED(1) ffff82d080a09000 (.data=ffff82d080a0a000, .rodata=ffff82d080a0a000) using 2 pages.
  (XEN)     do_xen_version patch ffff82d080115069(1930) with ffff82d080a09000 (1956)
  (XEN)     compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a097c8 (1879)
  (XEN) build-id=dbbd14b6aa110b9065e2667d42ef4291eed6b821
  (XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12
  (XEN)  name=test2 state=CHECKED(1) ffff82d080a0c000 (.data=ffff82d080a0d000, .rodata=ffff82d080a0d000) using 2 pages.
  (XEN)     compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0c000 (1887)
  (XEN)     do_xen_version patch ffff82d080115069(1930) with ffff82d080a0c783 (1964)
  (XEN) build-id=50364b9dbdb57e15f57bac237fce7fdfc37a94bd
  (XEN) depend-on=dbbd14b6aa110b9065e2667d42ef4291eed6b821
  (XEN)  name=test3 state=CHECKED(1) ffff82d080a0f000 (.data=ffff82d080a10000, .rodata=ffff82d080a10000) using 2 pages.
  (XEN)     do_xen_version patch ffff82d080115069(1930) with ffff82d080a0f000 (1958)
  (XEN)     compat_xen_version patch ffff82d08013faf5(1853) with ffff82d080a0f7ca (1881)
  (XEN) build-id=0f2d152953babf40b6635e2c80adcb3ced2b04d9
  (XEN) depend-on=34c63348019acbe8bab69808a6f8ad7618fa7c12

  # xen-livepatch apply test1
  # # Verify test1 is applied
  # xen-livepatch apply test2
  # # Verify test2 is also applied
  # xen-livepatch list
    ID                                     | status
   ----------------------------------------+------------
   test1                                   | APPLIED
   test2                                   | APPLIED
   test3                                   | CHECKED

  # xen-livepatch replace test3
  # # Verify test3 is applied and test1 and test2 are not
  # xen-livepatch revert test3
  # # Verify test3 is not applied
  
  # xen-livepatch unload test1
  # xen-livepatch unload test2
  # xen-livepatch unload test3

TODOs

v1 of patchset is in the Xen code base. We still have a list of TODOs and track it here.

OSSTest

Make OSSTest do regression tests. Xen 4.9

Toolstack

Things that need to be done in there:

  • xen-livepatch better smarts. Right now you need to do 'upload','check' and 'apply'. Three operations which could be folded in one - since the system admin probably want to have all of those done at once. The 'all' option does that.
  • Instead of using 'xen-livepatch' we should use 'xl'. That above mentioned logic could be part of 'xl' while 'xen-livepatch' is an accessory function for expert users?
  • If hypervisor is compiled without Live Patch the tool should print simply '"livepatch support not available in Xen'
  • If the build-id is wrong and one tries to apply the payload the message of 'hello failed with 1(Operation not permitted)' is lacking. We should have more details (and maybe a different error?). Or perhaps check the build-d as as being upload (however we may load them out of order). Perhaps have the toolstack check the build-id? How to do that against loaded patches (have them locally saved?)
  • Read build-id of the payloads files and display them - as well the dependencies. Perhaps even add a simple dependency checker.

livepatch-build--tools

  • Provide better error-code - saying what is unsupported if the user tries to do it.
  • Able to generate payloads that NOP functions.
  • Able to generate payloads only against .data sections.
  • Create payloads for ARM 32
  • Create payloads for ARM 64

Hypervisor

Currently the implementation has many TODOs implemented.

However we need to expand the hypervisor to include support for:

  • build_id support Done! (4.7)
  • The check for --build-id needs to deal with localized binutils. (4.7)
  • Make it work with .rodata changes only, or just inline changes. (4.8)
  • Need 'dl_sym' functionality in hypervisor to parse ELF payload. Do dynamic linking. (Ross) (4.7)
  • Lookup and insertion in the symbol table. Compute the proper offset and virtual address based on symbol name (<do_domctl>) (Ross) (4.7)
  • Need to either remove duplicate names from the symbol table or have the hypervisor implement proper ELF parsing - integrate the elf sections and parse that during runtime. (4.7)
  • Insertion of exceptions in the exception table. Also support resorting it. The hot-patch may have new exceptions that need to be dealt with. (4.7)
  • Checking the build-id against the hypervisor. (4.7)
  • Code to patch and revert the hot-patch (for simplicity we could boot Xen with 'cpus=1' so we don't have to worry about proper code to trigger patching) (4.7)
    • Support NMI? (or something similar?) (4.7)
    • Patching requires saving the initial state. That needs to be saved for revert. (4.7)
    • Patching needs allocation of memory, putting in the code at memory. (4.7)
    • Deal with proper page table types for the modification (ro for .data, x for .text, etc). (4.7)
    • Code to revert. (4.7)
  • Code to activate patching mechanism. On VMEXIT routine right before checking the do_softirq. Ditto for PV paths. Call the top patching mechanism - which can IPI other CPUs (or wait until all CPUs have come to the same point). (4.7)
  • Add functionality to execute ELF code. (4.8)
  • Patch with NOPs if .livepatch.funcs->new_size has 0. (4.8)
  • Seperate pages for .text; .rodata; .bss and .data (4.8)
  • Make it work under ARM32:Done! (4.8)
    • Support ARM32 ELF relocations
    • Support ARM32 virtual addresses
    • Support a similar to CPU flushing semantics as x86
    • Support NMI? (or something similar?)
  • Make it work under ARM64: (4.8)
    • Support ARM64 ELF relocations
    • Support ARM64 virtual addresses
    • Support a similar to CPU flushing semantics as x86
  • A better mechanism to "mask" NMIs during patching. The existing mechanism looses NMI if they have been sent and we don't have a mechanism to replay them. Note that this is also fixes alternative section patching. Could (like Linux) annotate handlers don't get patched.
  • Code to verify signature of the ELF payload.
    • Doing the verification - checking if any code that uses the to-be-patched-function is in use. Save what the virtual address for the functions is - and verify the stack when patching (this incidentally is what .livepatch_section structures can have).
  • tboot - will need to remeasure hypervisor when adding new code
  • SecureBoot - not sure how it works with Live Patch. Needs to be tested. Xen should verify payload - and in turn use SecureBoot services to verify?
  • Make .livepatch.funcs be an SHF_MERGE type with .sh_entsize having the size of the struct.
  • Make the usage of tools/symbols use common list model (similar to how alternative code is built)
  • When figuring out the new_addr and the name is <symbol>+<offset> - compute that.
  • The code for doing the patching - call from __do_softirq - can also be used for runtime microcode loading.
  • Make XENPF_get_symbol also include Live Patch symbols.