https://wiki.xenproject.org/api.php?action=feedcontributions&user=Dave.scott&feedformat=atomXen - User contributions [en]2024-03-28T21:25:25ZUser contributionsMediaWiki 1.31.3https://wiki.xenproject.org/index.php?title=XAPI_Design_and_Code_Review&diff=13889XAPI Design and Code Review2015-01-20T21:51:20Z<p>Dave.scott: Link to the design doc index</p>
<hr />
<div>The Xapi code repositories are on Github in the [https://github.com/xapi-project xapi-project] organisation.<br />
<br />
If you want to make a change to Xapi then you should:<br />
<br />
# if it's a bug: make a pull request with a code fix (maybe simply an experimental one for comments-- don't be shy!)<br />
# if it's a new feature or impacts multiple components: describe your proposed changes in markdown first in a [https://github.com/xapi-project/xapi-project.github.io/pulls design doc pull request]. This allows the general approach to be checked before you spend too much time coding everything up. Assuming everything is ok, you should also produce updates to the relevant technical documentation too, so they don't get out-of-sync.<br />
<br />
To understand the level of detail required of a design doc, have a look at the [http://xapi-project.github.io/design-docs/index.html Design Docs Index].<br />
<br />
In both cases (code changes and doc changes) you should expect prompt feedback. If nothing has happened for a week or so, feel free to comment on the pull request to ask if there has been any progress and/or send an email on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]<br />
<br />
If at any point you aren't sure what to do then ask on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_Design_and_Code_Review&diff=13888XAPI Design and Code Review2015-01-20T21:49:19Z<p>Dave.scott: </p>
<hr />
<div>The Xapi code repositories are on Github in the [https://github.com/xapi-project xapi-project] organisation.<br />
<br />
If you want to make a change to Xapi then you should:<br />
<br />
# if it's a bug: make a pull request with a code fix (maybe simply an experimental one for comments-- don't be shy!)<br />
# if it's a new feature or impacts multiple components: describe your proposed changes in markdown first in a [https://github.com/xapi-project/xapi-project.github.io/pulls design doc pull request]. This allows the general approach to be checked before you spend too much time coding everything up. Assuming everything is ok, you should also produce updates to the relevant technical documentation too, so they don't get out-of-sync.<br />
<br />
In both cases (code changes and doc changes) you should expect prompt feedback. If nothing has happened for a week or so, feel free to comment on the pull request to ask if there has been any progress and/or send an email on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]<br />
<br />
If at any point you aren't sure what to do then ask on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13887Category:XAPI Devel2015-01-20T21:46:17Z<p>Dave.scott: Remove the hopelessly out-of-date Xapi sprint stuff</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Concepts|<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Dispatch|Message forwarding]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13886Category:XAPI Devel2015-01-20T21:44:17Z<p>Dave.scott: Tunnelling doc seems out of place</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Concepts|<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Dispatch|Message forwarding]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Archived/Tunnelling&diff=13885Archived/Tunnelling2015-01-20T21:41:21Z<p>Dave.scott: Link to the Xapi technical docs</p>
<hr />
<div>Note: the Xapi docs have been updated here: http://xapi-project.github.io/xapi/design/tunnelling.html<br />
<br />
= Tunnelling API design =<br />
<br />
== Background ==<br />
<br />
To isolate network traffic between VMs (e.g. for security reasons) one can use VLANs. The number of possible VLANs on a network, however, is limited, and setting up a VLAN requires configuring the physical switches in the network. GRE tunnels provide a similar, though more flexible solution. This document proposes a design that integrates the use of tunnelling in the XenAPI. The design relies on the recent introduction of the Open vSwitch in XCP, and requires an Open vSwitch ([[OpenFlow]]) controller (further referred to as ''the controller'') to set up and maintain the actual GRE tunnels.<br />
<br />
We suggest following the way VLANs are modelled in the datamodel. Introducing a VLAN involves creating a Network object for the VLAN, that VIFs can connect to. The `VLAN.create` API call takes references to a PIF and Network to use and a VLAN tag, and creates a VLAN object and a PIF object. We propose something similar for tunnels; the resulting objects and relations for two hosts would look like this:<br />
<br />
<br />
<pre><br />
PIF (transport) -- Tunnel -- PIF (access) \ / VIF<br />
Network -- VIF<br />
PIF (transport) -- Tunnel -- PIF (access) / \ VIF<br />
</pre><br />
<br />
<br />
== XenAPI changes ==<br />
<br />
New `tunnel` class:<br />
* Fields:<br />
** `string` '''`uuid`''' (read-only)<br />
** `PIF ref` '''access_PIF'''` (read-only)<br />
** `PIF ref` '''transport_PIF'''` (read-only)<br />
** `(string -> string) map` '''`status`''' (read/write); owned by the controller, containing at least the key `active`, and `key` and `error` when appropriate (see below)<br />
** `(string -> string) map` '''`other_config`''' (read/write)<br />
* Messages:<br />
** `tunnel ref` '''`create`''' `(PIF ref, network ref)`<br />
** `void` '''`destroy`''' `(tunnel ref)`<br />
<br />
New fields in PIF class (automatically linked to the corresponding `tunnel` fields):<br />
* `PIF ref set` '''`tunnel_access_PIF_of`''' (read-only)<br />
* `PIF ref set` '''`tunnel_transport_PIF_of`''' (read-only)<br />
<br />
For clients to determine which network backend is in use (to decide whether tunnelling functionality is enabled) a key `network_backend` is added to the `Host.software_version` map on each host. The value of this key can be:<br />
* `bridge`: the Linux bridging backend is in use;<br />
* `openvswitch`: the [[Open vSwitch]] backend is in use.<br />
<br />
Notes:<br />
* The user is responsible for creating tunnel and network objects, associating VIFs with the right networks, and configuring the physical PIFs, all using the XenAPI/CLI/XC.<br />
* The `tunnel.status` field is owned by the controller. It may be possible to define an RBAC role for the controller, such that only the controller is able to write to it.<br />
* The `tunnel.create` message does not take a tunnel identifier (GRE key). The controller is responsible for assigning the right keys transparently. When a tunnel has been set up, the controller will write its key to `tunnel.status:key`, and it will set `tunnel.status:active` to `"true"` in the same field.<br />
* In case a tunnel could not be set up, an error code (to be defined) will be written to `tunnel.status:error`, and `tunnel.status:active` will be `"false"`.<br />
<br />
== XAPI ==<br />
<br />
`tunnel.create`:<br />
* Fails with OPENVSWITCH_NOT_ACTIVE if the Open vSwitch networking sub-system is not active (the host uses linux bridging).<br />
* Fails with IS_TUNNEL_ACCESS_PIF if the specified transport PIF is a tunnel access PIF.<br />
* Takes care of creating and connecting the new tunnel and PIF objects.<br />
** Sets a random MAC on the access PIF.<br />
** IP configuration of the tunnel access PIF is left blank. (The IP configuration on a PIF is normally used for the interface in dom0. In this case, there is no tunnel interface for dom0 to use. Such functionality may be added in future.)<br />
** The `tunnel.status:active` field is initialised to `"false"`, indicating that no actual tunnelling infrastructure has been set up yet.<br />
* Calls `PIF.plug` on the new tunnel access PIF.<br />
<br />
`tunnel.destroy`:<br />
* Calls `PIF.unplug` on the tunnel access PIF.<br />
* Destroys the `tunnel` and tunnel access PIF objects.<br />
<br />
`PIF.plug` on a tunnel access PIF:<br />
* Fails with TRANSPORT_PIF_NOT_CONFIGURED if the underlying transport PIF has `PIF.ip_configuration_mode = None`, as this interface needs to be configured for the tunnelling to work. Otherwise, the transport PIF will be plugged.<br />
* XAPI requests `interface-reconfigure` to "bring up" the tunnel access PIF, which causes it to create a local bridge.<br />
* No link will be made between the new bridge and the physical interface by `interface-reconfigure`. The controller is responsible for setting up these links. If the controller is not available, no links can be created, and the tunnel network degrades to an internal network (only intra-host connectivity).<br />
* `PIF.currently_attached` is set to `true`.<br />
<br />
`PIF.unplug` on a tunnel access PIF:<br />
* XAPI requests `interface-reconfigure` to "bring down" the tunnel PIF, which causes it to destroy the local bridge.<br />
* `PIF.currently_attached` is set to `false`.<br />
<br />
`PIF.unplug` on a tunnel transport PIF:<br />
* Calls `PIF.unplug` on the associated tunnel access PIF(s).<br />
<br />
`PIF.forget` on a tunnel access of transport PIF:<br />
* Fails with PIF_TUNNEL_STILL_EXISTS.<br />
<br />
`VLAN.create`:<br />
* Tunnels can only exist on top of physical/VLAN/Bond PIFs, and not the other way around. `VLAN.create` fails with IS_TUNNEL_ACCESS_PIF if given an underlying PIF that is a tunnel access PIF.<br />
<br />
Pool join:<br />
* As for VLANs, when a host joins a pool, it will inherit the tunnels that are present on the pool master.<br />
* Any tunnels (tunnel and access PIF objects) configured on the host are removed, which will leave their networks disconnected (the networks become internal networks). As a joining host is always a single host, there is no real use for having had tunnels on it, so this probably will never be an issue.<br />
<br />
== The controller ==<br />
<br />
* The controller tracks the `tunnel` class to determine which bridges/networks require GRE tunnelling.<br />
** On start-up, it calls `tunnel.get_all` to obtain the information about all tunnels.<br />
** Registers for events on the `tunnel` class to stay up-to-date.<br />
* A tunnel network is organised as a star topology. The controller is free to decide which host will be the central host ("switching host").<br />
* If the current switching host goes down, a new one will be selected, and GRE tunnels will be reconstructed.<br />
* The controller creates GRE tunnels connecting each existing Open vSwitch bridge that is associated with the same tunnel network, after assigning the network a unique GRE key.<br />
* The controller destroys GRE tunnels if associated Open vSwitch bridges are destroyed. If the destroyed bridge was on the switching host, and other hosts are still using the same tunnel network, a new switching host will be selected, and GRE tunnels will be reconstructed.<br />
* The controller sets `tunnel.status:active` to `"true"` for all tunnel links that have been set up, and `"false"` if links are broken.<br />
* The controller writes an appropriate error code (to be defined) to `tunnel.status:error` in case something went wrong.<br />
* When an access PIF is plugged, and the controller succeeds to set up the tunnelling infrastructure, it writes the GRE key to `tunnel.status:key` on the associated tunnel object (at the same time `tunnel.status:active` will be set to `"true"`).<br />
* When the tunnel infrastructure is not up and running, the controller may remove the key `tunnel.status:key` (optional; the key should anyway be disregarded if `tunnel.status:active` is `"false"`).<br />
<br />
== CLI ==<br />
<br />
New `xe` commands (analogous to `xe vlan-`):<br />
* `tunnel-create`<br />
* `tunnel-destroy`<br />
* `tunnel-list`<br />
* `tunnel-param-get`<br />
* `tunnel-param-list`<br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_Pool_Join&diff=13884XAPI Pool Join2015-01-20T21:32:55Z<p>Dave.scott: Add a note about truth and the source code</p>
<hr />
<div>== XAPI Pool Join ==<br />
This page partly describes the internal mechanism behind the pool join process.<br />
The ultimate source of truth is in the [[https://github.com/xapi-project/xen-api/blob/master/ocaml/xapi/xapi_pool.ml source code]]: this page simply gives a flavour of the kind of thing that goes on.<br />
<br />
The term "candidate host" is used to refer to the host which is attempting to join the pool.<br />
<br />
=== Preliminaries ===<br />
# Check that the candidate host does not have HA enabled. <br />
# Read the non-shared SRs from the local database. <br />
# Log in to the pool's master. <br />
# Check that HA isn't enabled in the pool. <br />
<br />
=== Pre-join assertions ===<br />
# Check that the candidate host is not attempting to join itself (assert_not_joining_myself). <br />
# Check that there are no other hosts in the local database (assert_i_know_of_no_other_hosts). <br />
# Check that there are no running or suspended VMs on the candidate host (assert_no_running_or_suspended_vms_on_me). <br />
# Check that there are no VMs with current operations (assert_no_vms_with_current_ops). <br />
# Check that hosts are homogeneous in terms of software version and CPU (unless the "force" option is used on the pool join operation) (assert_hosts_homogeneous). <br />
# Check that the candidate host has no shared SRs (assert_no_shared_srs_on_me). <br />
# Check that the management interface is (or contains) a physical NIC (assert_management_interface_is_physical). <br />
# Check that the external authentication type and service name matches the master (assert_external_auth_matches). <br />
<br />
=== Set up ===<br />
# Obtain the pool's "cluster secret" (Client.Pool.initial_auth). <br />
# Fetch a backup of the master's database (Pool_db_backup.fetch_database_backup). <br />
<br />
=== Synchronise state with the master ===<br />
<br />
This is done in a best-effort fashion; errors are suppressed. As much state as possible is synchronised.<br />
<br />
# Get the record for the candidate host (or create one none already exists) (create_or_get_host_on_master). <br />
# Create records for SR and PBD on master (SR.introduce; copy other-config over; PBD.create) (create_sr_and_pbd_on_master). <br />
# Create records for VMs on master (create_vm_on_master). <br />
# Remap VDIs to have SR fields matching the newly created SR records (remap_vdi). <br />
# Create records for VDIs on master (create_vdi_on_master). <br />
# Remap VBDs to have VDI/VM fields matching the newly created VDI records (remap_vbd). <br />
# Create records for VBDs on master (create_vbd_on_master). <br />
# Create physical PIFs (create_pif_and_maybe_network). Re-use the corresponding network if the master has a corresponding PIF. Otherwise create a new network. <br />
# Remap VIFs to have network fields matching the newly created Network records (remap_vif). <br />
# Create records for VIFs on master (create_vif_on_master). <br />
# Set the default crashdump SR and default suspend-image SR on the candidate host's host record. <br />
<br />
=== Finish-up ===<br />
# Write the cluster secret to the Xapi_globs.pool_secret_path file. <br />
# Set the candidate host's pool role to be a slave (Pool_role.set_role). <br />
# Restart xapi (Xapi_fuse.light_fuse_and_run). <br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_Dispatch&diff=13883XAPI Dispatch2015-01-20T21:30:42Z<p>Dave.scott: Add a little note about trusting this information</p>
<hr />
<div>== XAPI Service Dispatch and Message Forwarding ==<br />
This page partly describes the service dispatch and message forwarding mechanism in xapi. The authoritative source of wisdom remains the<br />
[https://github.com/xapi-project/xen-api source code].<br />
<br />
=== Incoming Call Dispatch ===<br />
In xapi/xapi.ml, HTTP servers are run on both a Unix domain socket and on an Internet port. An HTTP handler is added for incoming HTTP POST requests at URI "/":<br />
<br />
<br />
<pre><br />
Xapi_http.add_handler Http.Post "/" (Api_server.callback false)<br />
</pre><br />
<br />
<br />
The HTTP handler will thus invoke Api_server.callback for incoming XML-RPC requests. This function, in xapi/api_server.ml, parses the body.<br />
<br />
If the slave is in emergency mode and the call is not an emergency call, an error is raised. If this instance of xapi is not the pool master, and the call came in via the Unix domain socket, and the call is not a "login" call, then forward the request to the master using the "forward" function. Otherwise, Server.dispatch_xml is invoked:<br />
<br />
<br />
<pre><br />
let response = Server.dispatch_xml req fd xml<br />
</pre><br />
<br />
<br />
The dispatch_xml function (defined in autogen/server.ml) starts a new task to handle the call and compares the incoming call against a long list of potential calls. When a match is found, Server_helpers.do_dispatch is called, for example:<br />
<br />
<br />
<pre><br />
Server_helpers.do_dispatch ~session_id ~forward_op "<none/>" __async supports_async<br />
<br />
__call local_op marshaller fd http_req __label generate_task_for<br />
</pre><br />
<br />
<br />
The parameters to do_dispatch indicate a variety of things such as whether the call can be performed asynchronously, how the arguments are to be unmarshalled, etc. Also provided are a "local_op" and a "forward_op" which describe what to do if the call is to be serviced locally or forwarded, respectively.<br />
<br />
The do_dispatch function (defined in idl/ocaml_backend/server_helpers.ml) creates a new thread for servicing the call for those which are executed asynchronously. It then calls "exec_with_context" which in turn calls "exec" which executes the local_op if it's not the master or the forward_op if it is the master.<br />
<br />
The local_op specified in autogen/server.ml sometimes delves straight into the datamodel (e.g. via Db_actions.DB_Action, defined in autogen/db_actions.ml) but often refer to a function in Custom. This module is passed in via the Make functor and is instantiated with the Actions module from xapi/api_server.ml which merely translates sub-module names such as Host to the Xapi_host module whose implementation can be found in xapi/xapi_host.ml. Hence a mention of Custom.Host.disable_binary_storage refers to the implementation of the function "disable_binary_storage" in xapi/xapi_host.ml. These functions describe the logic requested by the call.<br />
<br />
The forward_op often makes reference to a function in Forward, which again is a module passed in via the Make functor, instantiated with the result of applying the Actions module from xapi/api_server.ml to the Message_forwarding.Forward functor, defined in xapi/message_forwarding.ml. Hence a mention of Forward.Host.disable_binary_storage refers to the implementation of the function "disable_binary_storage" in the Forward module in xapi/message_forwarding.ml, described in the next section.<br />
<br />
=== Message Forwarding ===<br />
<br />
In the case where Server_helpers.do_dispatch elects to execute the forward_op rather than the local_op, the request must be forwarded to the appropriate host, which may even be the localhost. This is dealt with by the message forwarding layer, implemented in xapi/message_forwarding.ml.<br />
<br />
The Forward functor contains a number of modules, such as Pool, Host, etc. Each module contains functions which take a context, a host, plus any arguments corresponding to the parameters of the API call. Typically, these functions invoke "do_op_on", passing both a "local_fn" and an "op":<br />
<br />
<br />
<pre><br />
let do_op_on ~local_fn ~__context ~host op = ...<br />
</pre><br />
<br />
<br />
The local_fn is usually a function from the Local module. This module is passed in via the Forward functor and is instantiated with the Actions module from xapi/api_server.ml (see above). Hence a mention of Local.Host.disable_binary_storage refers to the implementation of the function "disable_binary_storage" in xapi/xapi_host.ml.<br />
<br />
The op is a function which performs an RPC to the appropriate host.<br />
<br />
The do_op_on function checks that the target host is alive, otherwise raising a "host offline" server error then defers to the do_op_on_common function. This function checks whether the target host is actually the local host. If so, then the local_fn is executed. Otherwise, an expression such as the following is evaluated:<br />
<br />
<br />
<pre><br />
call_slave_with_session remote_rpc_retry __context host<br />
<br />
(set_forwarding_on_task context host)<br />
<br />
(fun session_id rpc -> Client.Host.disable_binary_storage rpc session_id host)<br />
</pre><br />
<br />
<br />
The function call_slave_with_session first logs in to the host, getting a session_id in return:<br />
<br />
<br />
<pre><br />
let session_id = Xapi_session.login_no_password ~__context ~uname:None ~host ~pool:true<br />
</pre><br />
<br />
<br />
It then determines the address of the host by querying the local database:<br />
<br />
<br />
<pre><br />
let hostname = Db.Host.get_address ~__context ~self:host<br />
</pre><br />
<br />
<br />
It then invokes the function passed in as the final argument, passing it the session_id and a function which performs the RPC to the remote host. An expression like the following is evaluated:<br />
<br />
<br />
<pre><br />
Client.Host.disable_binary_storage<br />
<br />
(remote_rpc_retry context hostname (set_forwarding_on_task context host))<br />
<br />
session_id host<br />
</pre><br />
<br />
<br />
The sub-expression "set_forwarding_on_task context host" finds a task relating to the given context in the database, and sets "forwarded" to true and "forwarded_to" to the host on it, returning the task if found.<br />
<br />
The sub-expression "remote_rpc_retry context hostname task_opt" is a function which takes an XML string (from the Xml.xml type) and performs the RPC using the XML-RPC library in idl/ocaml_backend/xmlrpcclient.ml.<br />
<br />
Hence the expression above boils down to:<br />
<br />
<br />
<pre><br />
Client.Host.disable_binary_storage (function xml -> [engp:send the XML-RPC request to the host]) session_id host<br />
</pre><br />
<br />
<br />
The implementation of the Client module can be found in autogen/client.ml. This uses the rpc_wrapper function to perform the RPC, generating the XML for the call with its arguments and parsing the return value.<br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13882Category:XAPI Devel2015-01-20T21:29:19Z<p>Dave.scott: Rename 'Dispatch' to 'Message fowarding' (the Xapi term)</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Concepts|<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Dispatch|Message forwarding]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[Tunnelling]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13881Category:XAPI Devel2015-01-20T21:27:53Z<p>Dave.scott: Rename "Designs" to "Concepts" which better describes content</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Concepts|<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[Tunnelling]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13880Category:XAPI Devel2015-01-20T21:25:30Z<p>Dave.scott: Remove hopelessly out-of-date description of startup</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[Tunnelling]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_Startup&diff=13879XAPI Startup2015-01-20T21:24:54Z<p>Dave.scott: Link to the code which is authoritative</p>
<hr />
<div>Note: this is out-of-date. To understand the Xapi startup, the code is the authoritative source: https://github.com/xapi-project/xen-api/blob/a232a3db5e51bf21eed2680386e4de79dcf4c70e/ocaml/xapi/xapi.ml#L690<br />
<br />
The source code controlling the xapi startup procedure is contained within api.hg/ocaml/xapi/xapi.ml. The majority of the startup code is contained within the server_init function.<br />
<br />
In the following sequence of steps, '''(both)''' denotes that the step is executed on both a pool master and a pool slave; (master) denotes that the step is only executed on a pool master; and '''(slave)''' denotes only on a slave.<br />
<br />
Where function names are specified, no module name implies that the function is in the Xapi module.<br />
<br />
'''(both) Parse command-line arguments'''<br />
<br />
See init_args.<br />
<br />
'''(both) Read config file'''<br />
<br />
See Helpers.read_config.<br />
<br />
'''(both) Initing stunnel path'''<br />
<br />
See Stunnel.init_stunnel_path.<br />
<br />
'''(both) Read log configuration file'''<br />
<br />
See Helpers.read_log_config.<br />
<br />
'''(both) Print "server starting" message'''<br />
<br />
See print_server_starting_message.<br />
<br />
'''(both) Parse inventory file'''<br />
<br />
See Xapi_inventory.read_inventory.<br />
<br />
'''(both) Initialise local database'''<br />
<br />
See init_local_database.<br />
<br />
'''(both) Read pool secret'''<br />
<br />
See Helpers.get_pool_secret.<br />
<br />
'''(both) Initialise licensing'''<br />
<br />
See License.initialise.<br />
<br />
'''(both) Output xapi version info to the log'''<br />
<br />
See show_config.<br />
<br />
'''(both) Check control domain'''<br />
<br />
See check_control_domain. We check whether the UUID of dom0 is the same as that in the inventory.<br />
<br />
'''(both) Set signal handlers'''<br />
<br />
See signals_handling.<br />
<br />
'''(both) Set up xenstore keys'''<br />
<br />
See xenstore_setup.<br />
<br />
'''(both) Initialise random number generator'''<br />
<br />
See random_setup.<br />
<br />
'''(both) Run startup check'''<br />
<br />
See startup_check. Check that xenstored, xenconsoled and blktapctrl processes are running.<br />
<br />
'''(both) Register http handlers'''<br />
<br />
See register_http_handlers.<br />
<br />
'''(both) Register RPC callbacks'''<br />
<br />
See register_rpc_callbacks.<br />
<br />
'''(both) Start listening on the Unix domain socket used for RPC.'''<br />
<br />
See listen_unix_socket.<br />
<br />
'''(both) Start listening on the localhost Internet socket used for RPC.'''<br />
<br />
See listen_localhost.<br />
<br />
'''(both) Start HA if configured'''<br />
<br />
See start_ha.<br />
<br />
'''(both) Setup DB configuration'''<br />
<br />
See setup_db_conf. Read from /etc/xensource/db.conf. Create a lock for each database connection. Set the database mode to indicate whether this is a master or a slave.<br />
<br />
'''(master) Start up database engine'''<br />
<br />
See start_database_engine.<br />
Create a new file if one does not already exist for each database connection (see Db_connections.maybe_create_new_db).<br />
Populate the in-memory cache from the connection containing the most recent generation of database (see Db_cache.DBCache.initialise_db_cache).<br />
Perform a garbage collection on the database in which records which are unreferenced are removed (see Db_gc.single_pass).<br />
Set all hosts to disabled, then signal the HA monitor thread that it can look at the database (see Xapi_ha.on_database_engine_ready).<br />
<br />
'''(both) Bring up the management interface'''<br />
<br />
See bring_up_management_if.<br />
<br />
'''(both) Register SR plugins'''<br />
<br />
See Sm.register.<br />
<br />
'''(both) Start periodic scheduler'''<br />
<br />
See Xapi_periodic_scheduler.loop. Starts a new thread which watches a queue of functions to execute at some time in the future. (See "Register periodic functions" below.)<br />
<br />
'''(slave) Log into the master (say hello to pool)'''<br />
<br />
Keep trying attempt_pool_hello until successful.<br />
<br />
'''(slave) Update in-memory cache'''<br />
<br />
See Db_cache.DBCache.initialise_db_cache. Populate the in-memory cache, potentially from a newer version of the database from the master.<br />
<br />
'''(both) Perform house-keeping on the internal database'''<br />
<br />
See Dbsync.setup. This invokes Dbsync_slave.update_env on all hosts and Dbsync_master.update_env on the master. This largely involves updating the database to reflect current state and synchronising the configurations between hosts across a pool.<br />
<br />
Executed on all hosts, Dbsync_slave.update_env does the following:<br />
<br />
* Create a localhost record if doesn't already exist <br />
* Record the hostname information in Xapi_globs.localhost_ref <br />
* Load the host RRD <br />
* Refresh host info fields (software version, etc) <br />
* Record host free memory in database <br />
* Copy license to database <br />
* Create host CPU record <br />
* Create domain 0 records <br />
* Resynchronise host crashdumps <br />
* Update VMs' states <br />
* Remove any leaked dom0 block-attached VBDs <br />
* Resynchronise bonded and VLAN PIF records with pool master <br />
* Resynchronise PIF.currently_attached <br />
* Check patch status (see Xapi_pool_patch.update_db) <br />
<br />
In addition, Dbsync_master.update_env does the following<br />
<br />
* Create root user <br />
* Create pool record <br />
* Set the master's IP in the database <br />
* Refresh the console URLs in the database <br />
* Reset all Halted VMs to a known good state <br />
* Cancel running tasks <br />
* Update the SM plugin table <br />
* Recreate any missing VLAN records <br />
* Create missing Host_metrics objects for all hosts in the pool <br />
* Create a PBD and SR for the Tools ISO. <br />
<br />
'''(master) Start database garbage collection thread'''<br />
<br />
Start a thread to periodically garbage collect records with hanging references in the database.<br />
<br />
'''(both) Check that current license is okay for pooling configuration'''<br />
<br />
Raise a system alert if the license does not support pooling and yet there exist slaves in the database (or this is a slave).<br />
<br />
'''(both) Set memory target for control domain'''<br />
<br />
See set_domain0_memory_target. Set the dom0 memory target in engp:[[XenStore]].<br />
<br />
'''(both) Start message e-mailing thread'''<br />
<br />
See Xapi_message.start_message_hook_thread. Starts a thread which executes /opt/xensource/libexec/mail-alarm for sending messages.<br />
<br />
'''(both) Start heartbeat thread'''<br />
<br />
See Db_gc.start_heartbeat_thread. Start heartbeat thread which periodically "tickles" the heartbeat.<br />
<br />
'''(both) Synchronise HA state'''<br />
<br />
See resynchronise_ha_state. If HA has been disabled on the pool while this host was offline but HA was previously enabled on it, then disable it locally. Alternatively, if HA has been enabled on the pool but this host is the master and has it disabled, then disable HA on the pool. If HA is enabled in the pool and this host is a slave and has it disabled, this is an inconsistent state, so switch to emergency mode.<br />
<br />
'''(master) Start pool database backup thread'''<br />
<br />
See Pool_db_backup.pool_db_backup_thread. Starts a thread to periodically flush the database to the slaves.<br />
<br />
'''(master) Register database change notifier'''<br />
<br />
See Xapi_event.register_hooks. Register Xapi_event.event_add as a callback for database events.<br />
<br />
'''(both) Start monitoring thread'''<br />
<br />
See Monitor.loop. On a new thread, periodically executes Monitor.do_monitor. This spits out database usage and CPU load information to the log (see Monitor_self.go), and updates the RRDs (see Monitor_rrds.update_rrds).<br />
<br />
'''(both) Start database calls monitoring thread'''<br />
<br />
See Monitor_dbcalls.monitor_dbcall_thread.<br />
<br />
'''(both) Start guest liveness monitoring thread'''<br />
<br />
See Xapi_guest_agent.guest_metrics_liveness_thread. Starts a thread which monitors the liveness of guests.<br />
<br />
'''(both) (Placeholder for touching a "ready" file)'''<br />
<br />
Currently does nothing, but can cause a file to be touched if set in Xapi_globs.ready_file.<br />
<br />
'''(master) Check that there are no other masters in the pool'''<br />
<br />
See check_no_other_masters. Check that there are no other masters in the pool by asking them all what they think they are. If any other thinks that they are a master, then this host reverts to being a slave, and restarts.<br />
<br />
'''(both) Register periodic functions'''<br />
<br />
See Xapi_periodic_scheduler.register. Adds functions to the periodic event scheduler to flush files to disk; rotate log files; garbage collect network bridges; show that the periodic event scheduler is still running; and backup RRDs.<br />
<br />
'''(both) Execute startup scripts'''<br />
<br />
See startup_script. Executes /opt/xensource/libexec/xapi-startup-script.<br />
<br />
'''(master) Execute scripts in the "master-scripts" directory with param "start"'''<br />
<br />
See Xapi_pool_transition.run_external_scripts. Invoke each script in /etc/xensource/master.d/ with the parameter start.<br />
<br />
'''(slave) Execute scripts in the "master-scripts" directory with param "stop"'''<br />
<br />
See Xapi_pool_transition.run_external_scripts. Invoke each script in /etc/xensource/master.d/ with the parameter stop.<br />
<br />
'''(master) Create networks'''<br />
<br />
See Create_networks.create_networks_localhost. Creates a guest installer network, on which guests have a private, local IP address.<br />
<br />
'''(both) Bring up physical NICs'''<br />
<br />
See Xapi_pif.start_of_day_best_effort_bring_up. Try to bring up physical PIFs (by calling Xapi_pif.plug for each), in a best-effort fashion.<br />
<br />
Xapi_pif.plug calls Xapi_network.attach which instantiate the network (i.e. bridge) associated with the PIF. This is achieved via Nm.bring_pif_up which calls the interface-reconfigure script with the "up" command unless the PIF is marked with currently_attached as true.<br />
<br />
'''(both) Initialise storage'''<br />
<br />
See Create_storage.create_storage_localhost. Creates PBDs for shared SRs.<br />
<br />
'''(both) Start events thread'''<br />
<br />
See Events.listen_xal.<br />
<br />
'''(master) Start SR scanning thread'''<br />
<br />
See Xapi_sr.scanning_thread. Start a thread to scan SRs.<br />
<br />
'''(master) Check / create templates'''<br />
<br />
See Create_templates.create_all_templates.<br />
<br />
'''(both) (Placeholder for touching an "initialisation complete" file)'''<br />
<br />
Currently does nothing, but can cause a file to be touched if set in Xapi_globs.init_complete.<br />
<br />
'''(both) Establish version of tools VDI'''<br />
<br />
See Xapi_pv_driver_version.get_latest_tools_vsn. Sets Xapi_globs.tools_version to the version of the currently-inserted XS tools ISO.<br />
<br />
'''(slave) Fetch database backup from master'''<br />
<br />
See Pool_db_backup.fetch_database_backup. Fetch the database from the master and flush it to each database connection.<br />
<br />
'''(both) Plug unplugged PBDs'''<br />
<br />
See plug_unplugged_pdbs. Calls Create_storage.plug_unplugged_pbds.<br />
<br />
'''(master) Consider sending a master transition alert'''<br />
<br />
If this node has just become a master then raise a system alert to spread this news.<br />
<br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13878Category:XAPI Devel2015-01-20T21:22:37Z<p>Dave.scott: Remove hopelessly out-of-date VM migration code walkthrough</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_VM_Migration&diff=13877XAPI VM Migration2015-01-20T21:21:47Z<p>Dave.scott: Link to the code which is authoritative</p>
<hr />
<div>Note: this description has been out of date for a while. To see how migration works, it is best to read the code starting from here: https://github.com/xapi-project/xen-api/blob/a232a3db5e51bf21eed2680386e4de79dcf4c70e/ocaml/xapi/message_forwarding.ml#L1638<br />
<br />
<br />
== XAPI VM Migration ==<br />
Here's an overview of how xapi does migration of VMs. The code can be found in '''api.hg/ocaml/xapi/xapi_vm_migrate.ml'''.<br />
<br />
{| class="prettytable"<br />
! '''Receiver''' <br />
|-<br />
| attach the VDIs <br />
|-<br />
| create the domain <br />
|-<br />
| restore the devices (unless we must delay until after VDI activation) <br />
|-<br />
| send handshake to transmitter <br />
|-<br />
| '''''Synchronisation point 1'''''<br />
|-<br />
| restore the domain, deserialising it from the transmitter <br />
|-<br />
| <br />
|-<br />
| '''''Synchronisation point 2'''''<br />
|-<br />
| receive handshake from transmitter <br />
|-<br />
| <br />
|-<br />
| '''''Synchronisation point 3'''''<br />
|-<br />
| activate VDIs (unless it's a localhost migrate) <br />
|-<br />
| restore the devices (unless done already) <br />
|-<br />
| unpause the domain <br />
|-<br />
| plug PCI devices <br />
|-<br />
| send handshake to transmitter <br />
|-<br />
| '''''Synchronisation point 4'''''<br />
|-<br />
| <br />
|}<br />
<br />
You may be interested in a proposal for [[CrossPoolMigration]]<br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13876Category:XAPI Devel2015-01-20T21:19:23Z<p>Dave.scott: Remove compress export proposal</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Compressing_VM_Exports&diff=13875Compressing VM Exports2015-01-20T21:18:25Z<p>Dave.scott: Link to the XenAPI reference where this is documented</p>
<hr />
<div>Note: this has been implemented and is documented in the [http://xapi-project.github.io/xen-api/importexport.html XenAPI reference]<br />
<br />
= Compressing VM exports =<br />
== Background ==<br />
There are two kinds of VM exports in XCP:<br />
# metadata only<br />
# full VM: metadata plus disk blocks<br />
An export is a tar format archive where the metadata is always the first file and has name "ova.xml".<br />
The metadata is stored in XML format and includes a bunch of existing version numbers:<br />
{| border="1" cellpadding="2" cellspacing="0"<br />
| key <br />
|-<br />
| product_version <br />
|-<br />
| build_number<br />
|-<br />
| xapi_major <br />
|-<br />
| xapi_minor <br />
|-<br />
| export_vsn <br />
|}<br />
The CLI commands involved are:<br />
<br />
<pre><br />
xe vm-export vm=... filename=...<br />
xe vm-import filename=...<br />
</pre><br />
<br />
Exports and imports can also be processed by HTTP GET and PUT.<br />
<br />
== Problem statement ==<br />
The "full VM" exports can be very large since they contain raw disk blocks. Their size makes them difficult to store and distribute over the network.<br />
<br />
== Proposal ==<br />
The disk blocks in a "full VM" export often contain very little user data and compress very well with gzip. I propose to:<br />
# add a --compress option to the CLI which will cause VM exports to be compressed via gzip<br />
# on import, auto-detect whether a VM requires decompression and decompress it first.<br />
<br />
=== Auto-detection algorithm ===<br />
Approximately:<br />
# Read the first 512 bytes (a tar header length)<br />
# Check if the tar header is well-formed and refers to an ova.xml file<br />
## f so: consider as uncompressed<br />
## If not: open a pipe to zcat, in a background thread write the first 512 bytes and then the rest of the file. Read from the output of zcat and look for a tar header<br />
<br />
=== Version number discussion ===<br />
The export format has two layers:<br />
# the outermost ("envelope"?) layer which currently contains: tar with an xml file containing version numbers<br />
# the innermost ("payload"?) layer which currently contains: the rest of the VM metadata and optional disk blocks<br />
The changes anticipated in the future all concern the "innermost" layer and hence are covered by the existing version numbers. For example I anticipate<br />
# adding new VM metadata<br />
# changing the disk block format e.g. to vhd<br />
Adding another version number to cover the outermost layer seems like overkill because:<br />
# it is simple enough to tell the difference between tar and tar.gz automatically<br />
# no future changes to this layer are anticipated<br />
# if we change the outermost layer then we'll simply extend the auto-detect code.<br />
<br />
== Implications ==<br />
Old (uncompressed) exports will continue to import on new servers, thanks to the auto-detection.<br />
<br />
New compressed exports will not import on old hosts. They will fail with a generic "IMPORT_ERROR" exception (probably containing the text 'Failure "int_of_string"'). A simple workaround is to uncompress the file first and then import it.<br />
<br />
[[Category:XAPI Devel]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13874Category:XAPI Devel2015-01-20T21:16:52Z<p>Dave.scott: Remove the out of date "inter domain comms" page</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Inter-domain_communication_for_XAPI&diff=13873Inter-domain communication for XAPI2015-01-20T21:16:04Z<p>Dave.scott: This page is out of date -- should be replaced with a link to docs for the "message-switch"</p>
<hr />
<div>Note: this page is out-of-date and should be replaced with a description of the "message-switch" and "vchan"; see https://github.com/xapi-project/xapi-project.github.io/issues/39<br />
<br />
In the future we will have<br />
# storage driver domains<br />
# network driver domains<br />
# qemu stub domain(s)<br />
# ... various other helper domains (e.g. possibly a domain running pygrub)<br />
<br />
The XCP toolstack already has several pieces including:<br />
# xapi: handling resource pools<br />
# squeezed: applying ballooning policy<br />
# xenopsd: (new) "babysitting" running VMs on a host<br />
# perfmon: monitors alerts based on performance measurements<br />
<br />
We currently use a set of ad-hoc protocols for inter-process communication including<br />
# XMLRPC<br />
# xenstore "rpc"<br />
# JSONRPC<br />
<br />
We desire to standardise on a particular messaging/RPC system which<br />
# is available everywhere and very easy to use from many different source languages<br />
# promotes location-transparency (so we don't suffer when a service is moved from dom0 to a domU)<br />
<br />
= Proposal: DBUS =<br />
<br />
[http://en.wikipedia.org/wiki/D-Bus DBUS] is a simple IPC system originally designed for graphical desktop environments. It supports<br />
# a message broker which can buffer messages and start receiving services on demand<br />
# an IDL supporting both basic types (string, int etc) and structured types (arrays, structs)<br />
# languages including: C, python, ocaml<br />
# location-transparency via a notion of a "well-known bus address" (like org.xen.foo) used to identify services<br />
<br />
It is also the system used by XCI.<br />
<br />
Useful links:<br />
# [http://live.gnome.org/DFeet d-feet]: a pygtk-based object inspector<br />
# http://wiki.meego.com/D-Bus/Overview: an overview, including instructions for using d-feet<br />
<br />
== Concrete example ==<br />
<br />
Consider a simplified XCP system containing the following services:<br />
# xapi: running in a domU, handling XenAPI calls from clients<br />
# xenopsd: running in dom0, performing start/shutdown/... on running VMs<br />
# storage: running in a domU i.e. a storage driver domain<br />
<br />
We would create a special bus for our communication (we wouldn't use the system or the session default buses)<br />
<br />
Each service would bind to: org.xen.xcp.servicename<br />
<br />
Each service would place its objects into path: org/xen/xcp/servicename/objectname<br />
<br />
TBD: Would we want to expose internal application "objects" (e.g. the VMs being managed by xenopsd or the VDIs within a storage driver domain) as individual objects, so they are introspectable? Or would that require that we broadcast too much internal state?<br />
<br />
We would want to specify IDL up-front for interfaces (even though it can be used dynamically e.g. with python)<br />
<br />
TBD: Are signals useful for us?<br />
<br />
TBD: Are there any default interfaces for heartbeating and diagnostics?<br />
<br />
<br />
[[Category:Security]]<br />
[[Category:XAPI Devel]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13872Category:XAPI Devel2015-01-20T21:14:16Z<p>Dave.scott: Rename 'Disaggregated Xapi' to 'protocols'</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Protocols|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_Design_and_Code_Review&diff=13871XAPI Design and Code Review2015-01-20T21:11:41Z<p>Dave.scott: Created page with "The Xapi code repositories are on Github in the [https://github.com/xapi-project xapi-project] organisation. If you want to make a change to Xapi then you should: # if it's a b…"</p>
<hr />
<div>The Xapi code repositories are on Github in the [https://github.com/xapi-project xapi-project] organisation.<br />
<br />
If you want to make a change to Xapi then you should:<br />
<br />
# if it's a bug: make a pull request with a code fix (maybe simply an experimental one for comments-- don't be shy!)<br />
# if it's a new feature or impacts multiple components: describe your proposed changes in markdown first in a [https://github.com/xapi-project/xapi-project.github.io/pulls design doc pull request]. This allows the general approach to be checked before you spend too much time coding everything up.<br />
<br />
In both cases (code changes and doc changes) you should expect prompt feedback. If nothing has happened for a week or so, feel free to comment on the pull request to ask if there has been any progress and/or send an email on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]<br />
<br />
If at any point you aren't sure what to do then ask on the [http://lists.xenproject.org/mailman/listinfo/xen-api xen-api mailing list]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13870Category:XAPI Devel2015-01-20T21:04:23Z<p>Dave.scott: Add "Design and Code review" to Developing section</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Design and Code Review|Design and Code review]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13869Category:XAPI Devel2015-01-20T20:52:34Z<p>Dave.scott: Link to the XenAPI online reference</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://xapi-project.github.io/xen-api/index.html XenAPI reference]<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13868Category:XAPI Devel2015-01-20T17:38:52Z<p>Dave.scott: Remove VM startup: it was hopelessly out of date</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13867Category:XAPI Devel2015-01-20T17:38:20Z<p>Dave.scott: Link to repositories</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [https://github.com/xapi-project/ Repositories]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VM_Startup&diff=13866VM Startup2015-01-20T17:37:08Z<p>Dave.scott: Link to the code, this is all very out of date</p>
<hr />
<div>Note: this is a transliteration of the source code into English. It is very out-of-date. Please read the code comments instead, starting here: https://github.com/xapi-project/xen-api/blob/25487aa89476026b9f9bdf5a196710a0d61e808b/ocaml/xapi/message_forwarding.ml#L1188<br />
<br />
== VM Startup ==<br />
This page documents the current sequence of steps that take place when a VM is started.<br />
<br />
Note that it is only the code-path for successful VM starts which is documented here. In some cases, failure paths involve attempts to reverse previous steps.<br />
<br />
=== On the pool master ===<br />
http://xenbits.xen.org/xapi/xen-api.hg?file/25e9038fce00/ocaml/xapi/message_forwarding.ml<br />
in function VM.start<br />
<br />
* Mark the VM as currently being started (with_vm_operation). <br />
** Check that the start operation is valid on this VM (Xapi_vm_lifecycle.assert_operation_valid). <br />
** Add the start operation to the VM's current-operations (Db.VM.add_to_current_operations). <br />
** Update the VM's allowed operations (Xapi_vm_lifecycle.update_allowed_operations). <br />
* Mark the VM's VBDs as currently being attached (with_vbds_marked). On each VBD: <br />
** Check that the attach operation is valid on this VBD (Xapi_vbd_helpers.assert_operation_valid). <br />
** Add the attach operation to the VBD's current-operations (Db.VBD.add_to_current_operations). <br />
** Update the VBD's allowed operations (Xapi_vbd_helpers.update_allowed_operations). <br />
* Mark the VM's VIFs as currently being attached (with_vifs_marked). On each VIF: <br />
** Check that the attach operation is valid on this VIF (Xapi_vif_helpers.assert_operation_valid). <br />
** Add the attach operation to the VIF's current-operations (Db.VIF.add_to_current_operations). <br />
** Update the VIF's allowed operations (Xapi_vif_helpers.update_allowed_operations). <br />
* Compute the 'memory overhead' of the VM so we know how much memory we need on a target host<br />
* Find a suitable host <br />
** Either consult the 'Workload balancing' service (interface in workload_balancing.ml)<br />
** Or use the default logic in Xapi_vm_helpers.choose_host_for_vm:<br />
*** Iterate through hosts, starting with the host the VM has affinity to (if any), then the other hosts in a random order (from a distribution biassed towards hosts with more memory) until one is found on which the VM can boot (Xapi_vm_helpers.choose_host_for_vm_no_wlb): <br />
*** Check that the VM has an appropriate number of VCPUs (validate_vcpus). <br />
*** Check that the VM's memory constraints are valid (validate_memory).<br />
*** Check that the VM's shadow multiplier is valid (validate_shadow_multiplier). <br />
*** Check that the VM can perform "after-crash" actions (validate_actions_after_crash). <br />
*** Check that the host is live (assert_host_is_live). <br />
*** Check that the host is enabled (assert_host_is_enabled). <br />
*** Check that the host can see the SRs for the VM's VDIs (and the VM's suspend-VDI if it's suspended) (assert_can_see_SRs). <br />
*** Check that the host has a PIF for the networks for the VM's VIFs. <br />
*** Check that if the VM wants to boot HVM then the host is HVM-compatible. <br />
*** Check that the host has enough memory for the VM. <br />
*** Check that using this host preserves the HA plan (Xapi_ha_vm_failover.assert_vm_placement_preserves_ha_plan). <br />
* Allocate the VM to the chosen host (allocate_vm_to_host). <br />
** Set the VM's last-booted record (Helpers.set_boot_record). <br />
** Set the VM as 'scheduled_to_be_resident_on' the chosen host (Db.VM.set_scheduled_to_be_resident_on). <br />
* Execute the vm-start operation on the host (see below). <br />
* Once the operation is complete: <br />
** Remove the vm-start operation from the host's current-operations (Db.Host.remove_from_current_operations). <br />
** Update the host's allowed operations (Xapi_host_helpers.update_allowed_operations). <br />
** Unset the VM's 'scheduled_to_be_resident_on' field. <br />
** Mark the VM's VIFs as currently being attached (with_vifs_marked). On each VIF: <br />
** Remove the attach operation from the VIF's current-operations (Db.VIF.add_to_current_operations). <br />
** Update the VIF's allowed operations (Xapi_vif_helpers.update_allowed_operations). <br />
** Mark the VM's VBDs as currently being attached (with_vbds_marked). On each VBD: <br />
** Remove the attach operation from the VBD's current-operations (Db.VBD.add_to_current_operations). <br />
** Update the VBD's allowed operations (Xapi_vbd_helpers.update_allowed_operations). <br />
** Mark the VM as no longer currently being started (with_vm_operation). <br />
** Remove the start operation from the VM's current-operations (Db.VM.add_to_current_operations). <br />
** Update the VM's allowed operations (Xapi_vm_lifecycle.update_allowed_operations). <br />
** Update the VM's VBDs' allowed operations (update_vbd_operations). <br />
** Update the VM's VIFs' allowed operations (update_vif_operations). <br />
* Create a message saying that the VM was started. <br />
* Push the RRD to the host (Monitor_rrds.push_rrd). <br />
<br />
=== On the pool slave ===<br />
http://xenbits.xen.org/xapi/xen-api.hg?file/25e9038fce00/ocaml/xapi/xapi_vm.ml<br />
<br />
* Serialised with other host-local VM operations (in the vm_lifecycle_op queue), perform the following steps: <br />
** Lock the VM. This prevents the background event thread noticing any domains or devices.<br />
** Check that VM.power_state=Halted in the database<br />
** Get the last_booted_record of the VM from the database (Helpers.get_boot_record). <br />
** If VM.ha_restart_priority is set then set VM.ha_always_run to true in the database<br />
** If the VM has no BIOS strings set then set "generic whitebox" BIOS strings in the database<br />
** Starts the VM in the paused state (Vmops.start_paused). <br />
*** Check VM parameters (Vmops.check_vm_parameter). <br />
**** Check that the VM has more than 0 and fewer than 64 VCPUs. <br />
**** Check that the device name of each of the VM's VBDs is unique. <br />
**** Check that the VM has more than a global minimum amount of memory (currently 16MiB)<br />
** Get the list of VBDs which were marked by the message forwarding layer<br />
*** Ask the host memory ballooning daemon to reserve memory (greater than VM.dynamic_min, up to VM.dynamic_max)<br />
*** Verify that the host actually supports HVM, if this is an HVM domain<br />
*** Create an empty xen domain <br />
*** Add per-domain xenstore entries e.g. platform/ flags and BIOS strings. Note no devices are connected yet.<br />
*** Set the domain ID in the database. <br />
*** Clear all VBDs' and VIFs' 'currently-attached', 'status-code' and 'status-details' fields (Vmops.clear_all_device_status_fields). <br />
*** Check to make sure a VDI record exists for each VBD (Vbdops.check_vdi_exists). <br />
*** Set the machine address size, if it's specified in the VM's 'other-config' field. <br />
*** Choose a "kernel" (Vmops.create_kernel). <br />
**** Either the domain is PV so we either use the VM.PV_kernel path in dom0 or we run pygrub to extract the kernel from the guest's filesystem<br />
**** Or the domain is HVM so we use hvmloader<br />
** Call the domain builder in libxenguest to prepare the domain's memory image<br />
*** Also set vCPU -> pCPU affinity<br />
*** Also set vCPU scheduling parameters: weight and cap<br />
*** Also set the flags for NX, viridian, PAE, ACPI, APIC, ACPI S3, ACPI S4<br />
** For each VDI, call the storage backend vdi_attach (and, if necessary, vdi_activate)<br />
*** Create xenstore entries for each configured vCPU (this enables PV guests to hotplug/unplug)<br />
*** Add VIFs (Vmops.create_vifs). <br />
**** Create the bridge for the VIF's network (Xapi_network.attach_internal). <br />
**** If the VIF is on the special "Guest installer network", associate the DHCP lease to the VIF (Xapi_udhcpd.maybe_add_lease). <br />
**** Write netfront/netback entries to xenstore<br />
**** Wait for the add UEVENT to be processed by /etc/xensource/scripts/vif called from udev<br />
** If the guest is PV, write pcifront/backend entries to xenstore for any configured PCI passthrough devices<br />
*** Mark the correct number of CPUs as initially online (NB only PV guests support hotplug)<br />
*** If guest is HVM, create a qemu-dm process. qemu-dm is told to emulate RTL8193 NICs for up to the first 8 VIFs, a USB tablet and a VNC framebuffer<br />
**** NB the commandline links the VIF MAC address, dom0 bridge name and dom0 tapX.Y device together. For each working VIF 2 interfaces end up on the bridge: one vifX.Y representing the PV drivers and one tapX.Y representing the emulated hardware<br />
**** NB qemu only listens on 127.0.0.1 for vnc. It writes the port number to xenstore and xapi is responsible for forwarding connections to it.<br />
** If guest is PV, check if other-config:disable_pv_vnc is not true and run a vncterm process to export the console pty over VNC<br />
*** NB vncterm only listens on 127.0.0.1 for vnc. It writes the port number to xenstore and xapi is responsible for forwarding connections to it.<br />
*** NB If other-config:disable_pv_vnc=true, no vncterm copy is ran.<br />
** Write the VM.memory_dynamic_min and VM.memory_dynamic_max to xenstore for the per-host ballooning daemon to use<br />
*** Update VM_metrcs.set_start_time in the database<br />
*** Set the VM's last-booted record (Helpers.set_boot_record). <br />
* Delete any guest metrics associated with the VM in the database (Xapi_vm_helpers.delete_guest_metrics). <br />
** Atomically clears VM.scheduled_to_be_resident_on and sets VM.resident_on<br />
** If domain is to be started paused, then set VM.power_state to Paused<br />
** If domain is to be started running, unpause domain and set VM.power_state to Running<br />
** Unlock the VM. <br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13865Category:XAPI Devel2015-01-20T17:34:15Z<p>Dave.scott: The RRDs thing is really an API example</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
* [[XAPI RRDs|Downloading VM and Host metrics as RDDs]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13864Category:XAPI Devel2015-01-20T17:33:05Z<p>Dave.scott: Move fake PV drivers into the Developing section: this is really for the dev-test use-case</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
* [[XAPI fake presence of PV drivers|Dev-test features without VM PV tools]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=XAPI_fake_presence_of_PV_drivers&diff=13863XAPI fake presence of PV drivers2015-01-20T17:31:25Z<p>Dave.scott: Faking the PV drivers is a development trick</p>
<hr />
<div>= Faking the presence of the PV drivers =<br />
<br />
xapi currently insists that PV drivers are present before allowing the operations:<br />
* suspend/resume<br />
* migrate<br />
* clean reboot/shutdown<br />
* disk, nic hotplug/unplug<br />
The vast majority of testing is performed with PV drivers and it is recommended that PV drivers are always installed.<br />
<br />
However, at the xen-level, plain HVM guests without PV drivers can usually still be suspend/resumed and migrated, although with reduced performance (e.g. in the case of migrate: longer VM downtime). It is convenient to "fake" the presence of PV drivers in order to test these features during development, without having to always install PV tools in VMs.<br />
<br />
The following script takes the VM's uuid and writes the magic xenstore keys to fake the presence of up-to-date PV drivers. It must be run on the same host as the VM.<br />
<br />
<pre><br />
#!/bin/sh<br />
<br />
set -e<br />
<br />
usage() {<br />
echo "$0 <VM uuid>"<br />
echo " -- fakes the presence of PV tools in <VM uuid>"<br />
echo " -- NB the VM must be either paused or running on localhost"<br />
}<br />
<br />
fake(){<br />
local uuid=$1<br />
domid=$(xe vm-list uuid=${uuid} params=dom-id --minimal)<br />
xenstore-write /local/domain/${domid}/attr/PVAddons/MajorVersion ${major} \<br />
/local/domain/${domid}/attr/PVAddons/MinorVersion ${minor} \<br />
/local/domain/${domid}/attr/PVAddons/MicroVersion ${micro} \<br />
/local/domain/${domid}/data/updated 1<br />
}<br />
<br />
uuid=$(xe vm-list uuid=$1 params=uuid --minimal)<br />
if [ $? -ne 0 ]; then<br />
echo "Argument should be a VM uuid"<br />
usage<br />
exit 1<br />
fi<br />
<br />
# use the PRODUCT_VERSION from here:<br />
. /etc/xensource-inventory<br />
<br />
major=$(echo ${PRODUCT_VERSION} | cut -f 1 -d .)<br />
minor=$(echo ${PRODUCT_VERSION} | cut -f 2 -d .)<br />
micro=$(echo ${PRODUCT_VERSION} | cut -f 3 -d .)<br />
<br />
# Check the VM is running on this host<br />
resident_on=$(xe vm-list uuid=${uuid} params=resident-on --minimal)<br />
if [ "${resident_on}" != "${INSTALLATION_UUID}" ]; then<br />
echo "VM must be resident on this host"<br />
exit 2<br />
fi<br />
<br />
power_state=$(xe vm-list uuid=${uuid} params=power-state --minimal)<br />
case "${power_state}" in<br />
running)<br />
fake $uuid<br />
;;<br />
paused)<br />
fake $uuid<br />
;;<br />
*)<br />
echo "VM must be either running or paused"<br />
exit 3<br />
;;<br />
esac<br />
</pre><br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Design Document]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13862Category:XAPI Devel2015-01-20T17:29:08Z<p>Dave.scott: Host plugins is really a "how to write a xapi plugin": add to developing section</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Host Plugins|Writing XAPI Plugins]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13861Category:XAPI Devel2015-01-20T17:27:02Z<p>Dave.scott: Generalise Compiling to Developing and add two other "Tasks" to it</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Developing|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Compile_the_XAPI_toolstack_from_source&diff=13860Compile the XAPI toolstack from source2015-01-20T17:23:25Z<p>Dave.scott: Comically out of date build instructions fixed (hopefully)</p>
<hr />
<div>There are 2 reasonably simple ways to build XAPI and related components:<br />
<br />
== via opam ==<br />
<br />
Opam is a source-based package manager which has a developer-friendly workflow. The following commands assume an Ubuntu host, update to match your distro of choice:<br />
<br />
<pre><br />
apt-get install opam -y<br />
</pre><br />
<br />
Next the dependencies of xapi can be installed:<br />
<br />
<pre><br />
sudo apt-get install $(opam install xapi -e ubuntu)<br />
</pre><br />
<br />
Finally xapi itself can be built:<br />
<br />
<pre><br />
opam install xapi<br />
</pre><br />
<br />
== via RPM ==<br />
<br />
The open-source XenServer "buildroot" has a convenient Makefile to build Xapi via RPM (and experimentally via .debs also):<br />
<br />
On a CentOS 6 machine:<br />
<br />
<pre><br />
yum install -y epel-release<br />
</pre><br />
<br />
After adding EPEL, install and set up mock:<br />
<br />
<pre><br />
yum install -y mock redhat-lsb-core<br />
</pre><br />
<br />
Mock will refuse to run as root. You must choose a non-privileged user to run mock as. Type the following as root:<br />
<br />
(Note select a <user> which isn't "mock" when typing the commands below)<br />
<br />
<pre><br />
useradd <user> -G mock<br />
passwd <user><br />
su - <user><br />
</pre><br />
<br />
You are now ready to clone the buildroot repository and build the packages:<br />
<br />
<pre><br />
git clone git://github.com/xenserver/buildroot.git<br />
cd buildroot<br />
<br />
./configure<br />
make<br />
</pre><br />
<br />
[[Category:XAPI Devel]]<br />
[[Category:Development Process]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13859Category:XAPI Devel2015-01-20T17:16:33Z<p>Dave.scott: Add a Compiling section (will add more developer workflow links later)</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
}}<br />
<br>{{Trail|Compiling|<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br />
<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13858Category:XAPI Devel2015-01-20T17:10:15Z<p>Dave.scott: Remove Xenopsd from the proposals section</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Xenopsd&diff=13857Xenopsd2015-01-20T17:09:27Z<p>Dave.scott: Note that Xenopsd has been implemented and documented</p>
<hr />
<div>Note: Xenopsd has been implemented and is documented here: http://xapi-project.github.io/xenopsd/architecture.html<br />
<br />
= The xenops daemon =<br />
<br />
This is an attempt to improve the architecture of the XCP toolstack by splitting *domain management* into a separate service, known as the *xenops daemon*. This should have the following advantages:<br />
# it helps untangle the domain management code from the rest of xapi (which will focus on multi-host resource pool management)<br />
# it forces us to be explicit about the interface xapi needs, making the future libxl port more predictable and likely to succeed<br />
# it forces us to be explicit about the interface xenopsd needs, preventing (for example) the accidental insertion of a expensive RPC to a remote host through an internal xapi interface in a hot codepath. This is currently a very easy mistake to make and a better architecture would make it harder.<br />
<br />
== Building from source ==<br />
<br />
The current development build instructions are [[Building Xenopsd]]<br />
<br />
== Design principles == <br />
<br />
# xenopsd should ignore domains that it is not managing. This will allow (eg) domains started with 'xl' to safely co-exist<br />
## xenopsd could therefore be told to manage one VM, like 'xenvm' (part of XCI)<br />
## xenopsd could therefore be told to manage lots of VMs, like 'xapi' currently does<br />
# xenopsd should be capable of servicing all guest requests without making remote RPCs. This will allow (eg) VMs to reboot in an XCP resource pool even when the master is offline.<br />
# xenopsd should not require xapi to be running (although it will require a storage service to be available in order to attach new disks)<br />
# xenopsd should have a "simulator" backend as well as a real "xen" backend to allow us to replace the current "XIU" simulation system which doesn't support all operations (e.g. VM migrate)<br />
# xenopsd should have a simulator-based test suite to allow us to test the component at build time.<br />
# xenopsd should support a level-triggered (rather than edge-triggered) event system, to allow clients (ie 'xapi') to resynchronise as quickly as possible. It will be assumed that clients always want to 'diff' their current state against the 'tip' state and that they don't care about the exact sequence of historical changes (those are left for the audit log).<br />
# xenopsd should be able to handle operations on managed VMs concurrently (rather than serially) and avoid operations on one VM having dependencies on other VMs (eg VM_suspend should not depend on a dom0 VBD_plug)<br />
# xenopsd should be aware of driver-domains; there shall be no hardcoded "domid = 0" anywhere in the code<br />
# xenopsd should support cancellation for all blocking operations<br />
<br />
== Design overview ==<br />
<br />
At the high-level, we'll have a JSON-rpc interface whose concrete IDL is created with rpc-light. This will support operations to:<br />
# add/modify/remove VM metadata<br />
# start/stop/suspend/resume/migrate VMs<br />
# blocking poll for events<br />
Operations received through the interface will either be processed immediately (eg an event poll) or be queued, with one queue per VM. This ensures that VM operations are handled sequentially. The queue contents will be introspectable and cancellable through the API.<br />
<br />
A thread pool with a configurable number of threads will take operations from the per-VM queues and execute them.<br />
<br />
Machine-affecting operations (e.g. domain create, invocations of libxenguest and, in the future, libxl) will be through a backend plugin. There will be two plugins initially:<br />
# xen: this will use the existing xapi domain management code<br />
# simulator: this will pretend to manage domains without actually generating any side-effects.<br />
The backend interface will be as simple as possible. The backend will be selectable at load-time.<br />
<br />
The "xen" backend will monitor the domains running on a host, and look for:<br />
# domain creations<br />
# domain shutdowns<br />
# xenstore device frontend/backend changes<br />
Any changes on a managed VM will be converted into special messages:<br />
# VM_check_state<br />
# VBD_check_state<br />
These messages will be put in the appropriate per-VM queue and will cause the higher-level logic to poll the current state of the device (which may have changed since the event was triggered) and<br />
# initiate reboots<br />
# cleanup device state (e.g. call the SMAPI vdi_detach)<br />
<br />
The "simulator" backend will support a backdoor "trigger" operation which will allow a test harness to inject events.<br />
<br />
[[File:xenopsd-internals.svg]]<br />
<br />
== Interfacing with xapi ==<br />
The XCP toolstack, 'xapi' will defer to 'xenopsd' for all VM lifecycle operations. We shall have the following principles:<br />
# when a VM.resident_on is set, "dynamic" VM properties (e.g. guest agent version) shall be owned by the specified host, updated on receipt of events from xenopsd<br />
# when a "static" VM property is set (e.g. name_label), xapi shall reset the VM metadata in xenopsd so the property change takes effect on next reboot, on recept of events from the master xapi<br />
# on any event error (whether event from xenopsd or event from xapi), the code shall resynchronise from scratch<br />
<br />
The XenAPI VM.start operation shall look as follows:<br />
# on the master:<br />
## acquire resources (e.g. choose a host with enough capacity)<br />
## set VM.scheduled_to_be_resident_on to the specified host<br />
## forward the XenAPI call<br />
# on the host:<br />
## import the latest version of the metadata into xenopsd (so it doesn't depend on the pool db)<br />
## call VM.start on xenopsd<br />
## set VM.resident_on<br />
## insert a barrier into the xenopsd event stream, to ensure that all generated events are processed before the XenAPI call returns.<br />
<br />
<br />
== Code status ==<br />
<br />
The stable version of xenopsd was merged into XCP 1.6 and lives here:<br />
<br />
https://github.com/xen-org/xen-api/tree/master<br />
<br />
An experimental development version lives here:<br />
<br />
https://github.com/djs55/xenopsd/tree/master<br />
<br />
[[Category:XAPI Devel]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13856Category:XAPI Devel2015-01-20T17:08:43Z<p>Dave.scott: Link to Xenopsd architecture</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [http://xapi-project.github.io/xenopsd/architecture.html Xenopsd architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13855Category:XAPI Devel2015-01-20T17:05:15Z<p>Dave.scott: Installing VMs more easily has been superseded by packer and vagrant</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Making_it_easier_to_install_VMs_on_XCP&diff=13854Making it easier to install VMs on XCP2015-01-20T17:04:14Z<p>Dave.scott: Superseded by supporting packer and vagrant</p>
<hr />
<div>Note: this has been superseded by adding support to [https://github.com/rdobson/packer-builder-xenserver packer] and [https://github.com/jonludlam/vagrant-xenserver vagrant]<br />
<br />
Installing a linux VM on XCP requires the following steps:<br />
# Clone a template<br />
# Add a CDROM or a VIF<br />
# Tweak some VM.other_config:parameters (e.g. to include the exact URL of an install repo)<br />
# Manipulate the VBD.bootable flags<br />
# Answer the VM installer's questions<br />
<br />
Ideally we would have<br />
* Fewer steps (more sensible defaults)<br />
* More tolerance to user error (e.g. if the network URL isn't perfect, use a heuristic to find the install files)<br />
<br />
Here's an example showing how to install CentOS6:<br />
* [[CentOS 6 Virtual Machine installation on Xen]]<br />
<br />
= Ideas for improvement =<br />
<br />
# Add a VIF to our default templates to the main management network. This would make network installation easier.<br />
# If no network install URL is provided, fall back to a known good one<br />
## Even better would be to use a tool like Debian netselect to choose a good one<br />
# If a network install URL is provided but it slightly wrong (typical failure would be to point at a subdirectory by mistake), attempt to auto-correct it<br />
# If a CDROM is attached, attempt to detect the install files and work out what kind of system it is (debianlike or rhlike)<br />
# Allow the user to request an auto-install via an answerfile or preseed file<br />
## Even better would be to serve-up default answerfiles from the system<br />
<br />
;Getting away from eliloader<br />
Some operating systems (Debian Lenny/Squeeze/Wheezy, Ubuntu ~L,M,N,O, OpenSuse) can be easily installed via pre-shipped initrd (netinst) and kernel. This will:<br />
# Reduce all operations with other-config:install-repository<br />
# Allow to install operating systems without internet connectivity between dom0 and internet (note: guests can have it, but dom0 no more), this will make isolation of management network more reliable (Someone ether though about attack on dom0 via bad initrd, f.e. very big to eat all dom0 disk space?).<br />
# Simplify steps to start VM<br />
# elilo can be used for other templates (like older rhel with initrd patching).<br />
<br />
;Allow local console for PV-guests.<br />
When user make xe vm-start for initially created vm from special template, we can do following:<br />
# disable vnc console for initial startup (one time)<br />
# run VM on host where command was entered (will not work with remote xe)<br />
# do xl console for VM domain.<br />
User expirience will be very nice: xe vm-install template=BoBoBo, xe vm-start, and user got console of installing virtial machine.<br />
<br />
<br />
= Measuring the improvements =<br />
<br />
# Count the number of characters you need to type to install a particular guest type (eg Debian 6.0). Lower is better.<br />
<br />
<br />
<br />
[[Category:XAPI Devel]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13853Category:XAPI Devel2015-01-20T17:01:39Z<p>Dave.scott: We only need 1 entry for storage motion rather than 4</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[Making it easier to install VMs on XCP|Easier VM Install]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13852Category:XAPI Devel2015-01-20T16:57:00Z<p>Dave.scott: Add link to architecture documents</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [http://xapi-project.github.io/getting-started/architecture.html Xapi architecture]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[CrossPoolMigrationv1|Cross Pool Migration v1]]<br />
* [[CrossPoolMigrationv2|Cross Pool Migration v2]]<br />
* [[CrossPoolMigrationv3|Cross Pool Migration v3]]<br />
* [[Making it easier to install VMs on XCP|Easier VM Install]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13851Category:XAPI Devel2015-01-20T16:54:18Z<p>Dave.scott: DIsk import/export APIs is not a proposal any more</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[CrossPoolMigrationv1|Cross Pool Migration v1]]<br />
* [[CrossPoolMigrationv2|Cross Pool Migration v2]]<br />
* [[CrossPoolMigrationv3|Cross Pool Migration v3]]<br />
* [[Making it easier to install VMs on XCP|Easier VM Install]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Proposal:_Disk_import/export&diff=13850Proposal: Disk import/export2015-01-20T16:53:28Z<p>Dave.scott: </p>
<hr />
<div>Note: incremental disk import/export APIs have been merged and are documented here: http://xapi-project.github.io/xen-api/snapshots.html<br />
<br />
== Proposal to improve disk import/export ==<br />
<br />
There are various ways to move a disk into or out of a system via the XenAPI, both "official" (ie. via an API) and "unofficial" (ie. by working around missing APIs). Many of the "official" ways to move disks around use non-standard and poorly-documented protocols and formats, which inhibits interoperability with other services such as CloudStack and OpenStack. The "unofficial" ways are often risky since they potentially conflict with running operations (e.g. vhd coalesce).<br />
<br />
Disk import/export is a fundamental ability of a virtualisation platform, and good support for it is demanded by all users: traditional server virt users need off-site incremental backup; cloud orchestration layers need to deploy images from central repositories; everyone benefits from being able to quickly move gold image disk volumes around.<br />
<br />
This document starts with an overview of what we currently have; followed by an analysis of use-cases we'd like to support (or improve our support for); followed by a set of principles to govern our designs; and finally by a proposed set of APIs and CLI commands.<br />
<br />
== What do we currently have? ==<br />
<br />
The following sections describe the current mechanisms, who is known to use them, advantages and disadvantages of each.<br />
<br />
=== HTTP GET raw disk contents ===<br />
(In XenServer 6.2 and above)<br />
<br />
# (Optionally) The client calls XenAPI Task.create if it wants to be able to tell if the operation succeeded or failed. This is only optional to allow quick uploads entirely over HTTP.<br />
# The client sends an authenticated HTTP PUT to /export_raw_vdi?vdi=(ref or uuid). If a the client called Task.create it can add a "task_id=task reference" query parameter or cookie. Authentication can be either<br />
## basic auth (convenient for commandline usage with wget/curl)<br />
## a pre-created session_id query parameter or cookie header<br />
# The server<br />
## if the authentication cannot be verified then returns an HTTP 403 forbidden<br />
## if the VDI query parameter (or cookie) is not present then an uncaught exception causes the server to return HTTP 500 Internal server error<br />
## if the VDI ref or uuid doesn't exist then an uncaught exception causes the server to return HTTP 500 Internal server error<br />
## if the VDI is only accessible on a remote host then an HTTP 302 redirect is returned using the Host.address field (this won't work if a client is behind a NAT)<br />
## If all looks ok, the server returns HTTP 200 OK with headers<br />
### content-type: application/octet-stream<br />
### connection: close<br />
### task-id: XenAPI task ID<br />
# The server writes the unencoded disk contents<br />
# The server closes the connection at the end<br />
# (Optionally) The client waits until Task.get_finished is true, and then checks the value of Task.get_status to find out whether the operation succeeded or failed. Reasons the task may fail include<br />
## I/O errors reading from the backend substrate<br />
# If the client called Task.create then it now calls Task.destroy<br />
<br />
There is no xe CLI command.<br />
<br />
Advantages:<br />
# simple, can be driven entirely through a wget/curl invocation (without error reporting)<br />
<br />
Disadvantages:<br />
# missing CLI command<br />
# only supports raw format images<br />
# no support for downloading deltas<br />
# doesn't allow an aborted transfer to be resumed<br />
<br />
=== HTTP PUT raw disk contents ===<br />
<br />
# (Optionally) The client calls XenAPI Task.create if it wants to be able to tell if the operation succeeded or failed. This is only optional to allow quick uploads entirely over HTTP.<br />
# The client sends an authenticated HTTP PUT to /import_raw_vdi?vdi=(ref or uuid). If a the client called Task.create it can add a "task_id=task reference" query parameter or cookie. Authentication can be either<br />
## basic auth (convenient for commandline usage with wget/curl)<br />
## a pre-created session_id query parameter or cookie header<br />
# The server<br />
## if the authentication cannot be verified then returns an HTTP 403 forbidden<br />
## if the VDI query parameter (or cookie) is not present then an uncaught exception causes the server to return HTTP 500 Internal server error<br />
## if the VDI ref or uuid doesn't exist then an uncaught exception causes the server to return HTTP 500 Internal server error<br />
## if the VDI is only accessible on a remote host then an HTTP 302 redirect is returned using the Host.address field (this won't work if a client is behind a NAT)<br />
## If the client requests any HTTP transfer-encoding, the server returns HTTP 403 forbidden<br />
## If all looks ok, the server returns HTTP 200 OK with headers<br />
### content-type: application/octet-stream<br />
### connection: close<br />
### task-id: XenAPI task ID<br />
# The client<br />
## writes the unencoded disk contents<br />
## closes the connection at the end<br />
# (Optionally) The client waits until Task.get_finished is true, and then checks the value of Task.get_status to find out whether the operation succeeded or failed. Reasons the task may fail include<br />
## insufficient space in the VDI for the data provided<br />
## I/O errors writing to the backend substrate<br />
# If the client called Task.create then it now calls Task.destroy<br />
<br />
This can all be driven through the xe CLI command:<br />
<br />
xe vdi-import uuid=<target VDI> filename=<raw image><br />
<br />
This command takes care of authentication, Task handling and error reporting.<br />
<br />
Advantages:<br />
# simple, can be driven entirely through a wget/curl invocation (without error reporting) or via the CLI<br />
<br />
Disadvantages:<br />
# only supports raw format images<br />
# no support for uploading deltas<br />
# requires you to pre-create an image of the right size<br />
# doesn't allow an aborted transfer to be resumed<br />
# CLI has no progress monitoring<br />
<br />
=== HTTP PUT with 'chunked' encoding ===<br />
<br />
This is a version of the HTTP PUT with raw encoding with the following differences:<br />
# The client's HTTP PUT request contains the key "chunked" in either a query parameter or a cookie<br />
# The client sends the data in a stream of chunks, consisting of a header followed by a payload<br />
<br />
The header format is:<br />
# 64-bit little-endian: offset: the offset within the target disk in bytes<br />
# 32-bit little-endian: length: the number of bytes of data in this 'chunk'<br />
The length field measure the length of the payload which follows the header.<br />
<br />
The stream is terminated by a single chunk with both offset and length set to 0.<br />
<br />
There is no support in the CLI. This is known to be used by XenDesktop.<br />
<br />
Advantages:<br />
# allows sparse disks (containing big holes full of zeroes) to be uploaded efficiently<br />
<br />
Disadvantages:<br />
# upload only<br />
# no ability to resume an interrupted download<br />
# no CLI support<br />
# non-standard protocol, only available documentation seems to be this wiki page (!) after previous page was deleted<br />
# the only opportunity for error reporting is at the end<br />
<br />
=== Network Block Device (NBD) access ===<br />
<br />
This applies to XenServer 6.2 and later.<br />
<br />
For all the HTTP requests described below, authentication can be either<br />
# basic auth (convenient for commandline usage with wget/curl)<br />
# a pre-created session_id query parameter or cookie header<br />
<br />
For a given VDI uuid $VDI containined within SR uuid $SR:<br />
<br />
# The client generates a fresh $UUID<br />
# The client calls the internal storage API VDI.attach by<br />
## making an HTTP POST to /services/SM<br />
## containing an XMLRPC request. TODO: generate documentation for this.<br />
# The client calls the internal storage API VDI.activate by<br />
## making an HTTP POST to /services/SM<br />
## containing an XMLRPC request. TODO: generate documentation for this.<br />
# The client makes an HTTP PUT to /services/SM/nbd/$SR/$VDI/$UUID<br />
# The server replies with HTTP 200 OK including the header<br />
## Transfer-encoding: nbd<br />
<br />
At this point the client is connected to an NBD server and can access the disk.<br />
<br />
To finalise:<br />
# The client closes the NBD connection<br />
# The client calls the internal storage API VDI.deactivate by<br />
## making an HTTP POST to /services/SM<br />
## containing an XMLRPC request. TODO: generate documentation for this<br />
# The client calls the internal storage API VDI.detach by<br />
## making an HTTP POST to /services/SM<br />
## containing an XMLRPC request. TODO: generate documentation for this<br />
<br />
<br />
Advantages:<br />
# The NBD protocol is well supported by Linux and BSD<br />
# The NBD protocol is used by qemu/KVM for easy interop<br />
# The NBD protocol is understood by wireshark for easy debugging<br />
<br />
Disadvantages:<br />
# The current "wrapping" of the protocol is barking mad:<br />
## It exposes internal storage APIs across host, holding back storage API evolution<br />
## It requires the client to clean up everything at the end: the tapdisk resource left lying around until VDI.detach is called<br />
<br />
=== VM import/export ===<br />
<br />
We support 2 VM export formats:<br />
1. "geneva": this was the primary export format before XS 4.0 and should not be generated any more. We only support it just in case very old VM exports are still around. It will not be documented here.<br />
2. "xva": this is the primary export format used in XS 4.0 and later<br />
<br />
A "rio" export format consists of a tar file containing VM metadata and disk blocks. Example contents of a VM export:<br />
<br />
$ tar -tvf test.xva<br />
---------- 0/0 17391 1970-01-01 01:00 ova.xml<br />
---------- 0/0 1048576 1970-01-01 01:00 Ref:946/00000000<br />
---------- 0/0 40 1970-01-01 01:00 Ref:946/00000000.checksum<br />
---------- 0/0 1048576 1970-01-01 01:00 Ref:946/00000007<br />
---------- 0/0 40 1970-01-01 01:00 Ref:946/00000007.checksum<br />
---------- 0/0 1048576 1970-01-01 01:00 Ref:949/00000000<br />
---------- 0/0 40 1970-01-01 01:00 Ref:949/00000000.checksum<br />
---------- 0/0 1048576 1970-01-01 01:00 Ref:949/00000003<br />
---------- 0/0 40 1970-01-01 01:00 Ref:949/00000003.checksum<br />
<br />
The ova.xml contains a set of records, one per object in the xapi database. For a VM it will include VBDs, VDIs, SRs, VIFs, Networks. If requested at export time it will also include details of VM snapshots, their VBDs, VDIs etc. The objects are linked together by references (Ref:xxx). The disk content corresponding to VDI with reference Ref:946 will be in a subdirectory called "Ref:946" (see the listing above).<br />
<br />
The disk data is stored in a sequence of chunks. The size of each chunk is arbitrary but the common size is 1MiB. The chunks are ordered in a sequence, and the filename encodes a *sequence number* (not a disk offset). The first block is always present and given sequence number 00000000. Gaps in the sequence number space signal the presence of empty blocks, each the same size as the first block. For example, in the Ref:946 disk above, there is a gap of 6 between block 00000000 and 00000007, signalling the presence of 6 empty blocks of size 1MiB. The last block is always present to give a reader confidence that the stream has not been truncated.<br />
<br />
Each block is followed by a checksum. For a block with filename 00000000, the checksum would be 00000000.checksum. The checksum file contains the SHA1 of the disk contents in the corresponding data block, printed as a hex string i.e. the same output as given by<br />
<br />
$ sha1sum Ref\:946/00000000 | cut -f 1 -d " "<br />
3b71f43ff30f4b15b5cd85dd9e95ebc7e84eb5a3<br />
<br />
Note it is legal to insert empty data blocks into the stream, which consume a sequence number and which have a file_size of 0. Exporters may wish to insert these empty blocks if they are taking the time to scan the outgoing data for blocks of zeroes, in order to keep the data flowing and to prevent a network middlebox from silently closing the (apparently idle) connection.<br />
<br />
=== HTTP BITS via transfer VM ===<br />
<br />
TODO: fill this in later<br />
<br />
=== vhd manipulation through plugins ===<br />
<br />
OpenStack installs "plugins" in domain 0 and invokes these via the XenAPI. The plugins directly manipulate .vhd files contained within local "ext" and remote "nfs" SRs by:<br />
# moving new files in<br />
# updating the parent locator to create chains<br />
<br />
Advantages:<br />
# the plugins can be developed separately to XenServer<br />
<br />
Disadvantages:<br />
# the plugins run concurrently with coalesce and GC, potentially corrupting vhd chains or losing data<br />
# the plugins typically only support the file-based SR types, and leave out LVM. Supporting LVM is genuinely quite difficult and getting it wrong can cause data loss (e.g. through truncated device mapper entries)<br />
<br />
== Desired use-cases ==<br />
<br />
From [https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/msg00063.html Mate Lakat on xs-devel]:<br />
<br />
* Full disk input/output<br />
** User wants to stream a virtual disk to OpenStack glance quickly and efficiently, where Glance acts as an HTTP server<br />
** User wants to fetch a virtual disk from OpenStack glance quickly and efficiently, where Glance acts as an HTTP server<br />
<br />
Regarding "quickly and efficiently", this was explained by [https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/msg00074.html John Garbutt on xs-devel]:<br />
<br />
A key requirement is to not use large amounts of memory or disk space<br />
(10s of MB at most), when exporting disks up to about 1.5 TB in size,<br />
where they might need to be tarballed, then gzipped, before being set<br />
to a client. Thats why I suggested a bash pipe, but doesn't feel like<br />
the best approach.<br />
<br />
* Incremental backup/restore (for regular server virt as well as CloudStack)<br />
** User wants to make a full copy of a virtual disk (e.g. weekly)<br />
** User wants to take frequent (e.g. daily) incremental snapshots of the disk<br />
** User wants to export only the snapshot deltas (to save bandwidth)<br />
** User wants to limit the number of snapshots present (to save space and chain length, on both client and server)<br />
** User wants to import a full snapshot plus a set of deltas, to recover the disk<br />
<br />
* User wants to pipe a virtual disk through a bit-torrent client (or other third-party transcoding software) (see [https://lists.xenserver.org/sympa/arc/xs-devel/2013-08/msg00080.html John Garbutt on xs-devel])<br />
<br />
== Principles ==<br />
<br />
This section proposes some guiding principles to help guide the shape of the API.<br />
<br />
We shall<br />
# continue to support all currently supported protocols and formats.<br />
# default to generating export disk images in a well-known format with good documentation and sample code.<br />
## We could either invent a format and spend enough time to make it "well-known" or we could use someone else's format and share the documentation burden with them<br />
# work across all XenServer SR types, whether they happen to use our well-known format (e.g. vhd) or not<br />
## NB the format we choose to export and import in may be different to the common runtime format, since export/import may be optimised for readability while runtime will be optimised for random access performance. This is ok as long as efficient streaming conversion is possible.<br />
# allow access to disk data using well-known protocols with good documentation and sample code.<br />
# support both streaming input/output and random-access to/from external hosts<br />
## when streaming we shall support resumption of interrupted transfers<br />
# maximise efficiency by keeping the datapath short; by avoiding unnecessary copying; by using aligned I/O everywhere.<br />
# keep it simple by hiding internal XenServer interfaces from clients.<br />
# provide CLI wrappers around all of our lower-level APIs.<br />
<br />
== API changes ==<br />
<br />
We will continue to use the same authentication strategy for HTTP requests i.e.<br />
* we will accept basic auth for convenient commandline use with wget/curl<br />
* we will accept pre-allocated session_id parameters via query parameters or cookies<br />
<br />
We will build upon the concept of a *content_id* first introduced for storage migration. Unlike a VDI's uuid which uniquely identifies the storage *container* whose data constantly changes, the *content_id* identifies the state of the data blocks at a point in time. When a snapshot is taken the content_id will remain the same since the content is now read/only. When a disk is written to the content_id is changed to a unique value. If two VDIs have the same content_id then, irrespective of their heritage and tree structure, we know they are equivalent.<br />
<br />
During an HTTP download we will allow the client to request a range of bytes, in order to resume an aborted download.<br />
<br />
=== HTTP full disk download ===<br />
* Client:<br />
** HTTP GET /SM/sr-uuid/vdi-uuid<br />
* Server:<br />
** Accept-Ranges: bytes<br />
*** indicating that byte range requests are acceptable<br />
** Content-Disposition: attachment; filename=content_id.vhd<br />
*** encouraging the client to set the filename to the content_id, rather than the uuid. This is needed to handle chains properly, see below.<br />
** Streams out data as a consolidated dynamic .vhd file<br />
*** NB .vhd can be constructed on the fly: there is no need to use large amounts of temporary memory or space on the server.<br />
<br />
For convenience this will be wrapped in the xe command:<br />
<br />
xe vdi-export uuid=<vdi-uuid> --progress<br />
<br />
=== HTTP delta disk download ===<br />
To download the differences which would need to be applied on top of vdi-uuid2 (with content_id2) in order to end up with the same content as vdi-uuid (with content_id):<br />
<br />
* Client:<br />
** HTTP GET /SM/sr-uuid/vdi-uuid/differences-from/sr-uuid2/vdi-uuid2<br />
* Server:<br />
** Accept-Ranges: bytes<br />
** Content-Disposition: attachment; filename=content_id.vhd<br />
** Streams out a .vhd differencing disk with the parent locator set to "content_id2.vhd"<br />
<br />
For convenience this will be wrapped in the xe command:<br />
<br />
xe vdi-export uuid=<vdi-uuid> differences-from=<vdi-uuid2> --progress<br />
<br />
=== Coalesce backups ===<br />
<br />
Coalescing needs to happen in two places:<br />
1. on the XenServer<br />
2. on the machine handling the backups<br />
<br />
For the XenServer case, a coalesce is triggered as a background task when snapshots are deleted. The user needs to select which snapshots to delete based on their preferences, which could be<br />
1. delete all snapshots older than 'n' days<br />
2. delete all snapshots except the newest 'm'<br />
<br />
The low-level interface is simply<br />
<br />
xe vdi-destroy uuid=...<br />
<br />
TODO: do we need to provide extra help in the form of policy-specific high-level commands?<br />
<br />
To delete all snapshots older than 'n' days we could use a CLI command:<br />
<br />
xe vdi-destroy snapshot-of=<vdi uuid> snapshot-time="older than n days" --multiple<br />
<br />
To delete all snapshots except the newest 'm'<br />
<br />
TODO: figure out something reasonable here<br />
<br />
For the machine handling the backups, we'll provide a simple command-line tool which will coalesce a snapshot into its parent eg<br />
<br />
vhd-tool coalesce ...<br />
<br />
=== HTTP full/delta disk upload ===<br />
<br />
A full disk is a base dynamic .vhd plus zero or more differencing .vhds. The full disk is identified by the last delta disk.<br />
<br />
We shall support a low-level HTTP primitive:<br />
<br />
* Client:<br />
** HTTP PUT /SM/sr-uuid<br />
** Content-type: application/vhd<br />
** Client uploads the first few kb of the .vhd, which includes the header and backup footer<br />
* Server:<br />
** HTTP/201 Created<br />
** Location: /SM/sr-uuid/vdi-uuid<br />
<br />
The server will detect whether the .vhd is a dynamic vhd or a differencing .vhd. If it's a differencing .vhd it can create the new VDI as a snapshot of the one given by the content_id in the parent locator.<br />
<br />
If the .vhd is a differencing .vhd but the parent locator cannot be found then an HTTP/404 error code will be returned.<br />
<br />
For convenience this will be wrapped in the xe command:<br />
<br />
xe vdi-import filename=foo.vhd<br />
<br />
If foo.vhd is a differencing .vhd then the CLI will upload the base disks first, snapshot those and continue.<br />
<br />
=== Attaching VDIs to remote hosts ===<br />
<br />
We shall support NBD over pre-authenticated connections, but without clients having to call SMAPI VDI.{attach,activate,deactivate,detach} functions.<br />
<br />
* the Client:<br />
** HTTP CONNECT /SM/sr-uuid/vdi-uuid<br />
** Accept: nbd<br />
* the Server:<br />
** HTTP/200 OK<br />
* both client and server can exchange NBD messages<br />
* when the client closes the connection to the server then all the data is cleaned up<br />
<br />
=== VM import/export ===<br />
<br />
We shall replace the ad-hoc 'chunked' encoding with .vhd in the export format. We will continue to support both 'chunked' and 'vhd' for input.<br />
<br />
<br />
=== Open issues ===<br />
<br />
The focus here is on shifting the bits around. We've not discussed uploading/downloading/backing up VDI metadata (name-label, other-config keys etc)</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XAPI_Devel&diff=13849Category:XAPI Devel2015-01-20T16:50:46Z<p>Dave.scott: Remove experimental disk import/export APIs -- documented elsewhere</p>
<hr />
<div>{{TODO|Articles under '''Proposals''' need to be reviewed and if they are completed should be moved to '''Designs'''.}}<br />
----<br />
{{Trailbox|XAPI Developer Docs|<br />
|{{Trail|XAPI Project|<br />
* [http://xenproject.org/developers/teams/xapi.html Project Portal]<br />
* [[:Category:XAPI Sprints|XAPI Roadmap/Sprints]]<br />
* [[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]<br />
* [[Compile_the_XAPI_toolstack_from_source|Compile XAPI]]<br />
}}<br>{{Trail|Designs|<br />
* [[XAPI Adding Field|Adding Fields to XAPI]]<br />
* [[XAPI Adding Function|Adding Functions to XAPI]]<br />
* [[XAPI Dispatch|Dispatch]]<br />
* [[XAPI fake presence of PV drivers|Fake PV driver presence]]<br />
* [[XAPI Host Plugins|Host Plugins]]<br />
* [[OpaqueRef and uuid relationship in xapi|OpaqueRef and uuid]]<br />
* [[XAPI Pool Join|Pool Join]]<br />
* [[XAPI RRDs|RDDs]]<br />
* [[XAPI Startup|Startup]]<br />
* [[Tunnelling]]<br />
* [[XAPI VM Migration|VM Migration]]<br />
* [[VM Startup]]<br />
}}<br />
|{{Trail|Proposals|<br />
* [[VAAI support]]<br />
* [[dm-thin for local storage]]<br />
* [[Proposal: Disk import/export]]<br />
* [[Compressing VM Exports]]<br />
* [[CrossPoolMigration|Cross Pool Migration]]<br />
* [[CrossPoolMigrationv1|Cross Pool Migration v1]]<br />
* [[CrossPoolMigrationv2|Cross Pool Migration v2]]<br />
* [[CrossPoolMigrationv3|Cross Pool Migration v3]]<br />
* [[Making it easier to install VMs on XCP|Easier VM Install]]<br />
* [[NIC Bonding Improvements|NIC Bonding]]<br />
* [[XCP, RBAC and PAM authentication in XenAPI|RBAC, PAM authentication]]<br />
* [[Xapi on debian user experience improvements|XAPI on Debian Usability]]<br />
* [[XCP PV templates start|Starting PV Templates]]<br />
}}<br>{{Trail|Disaggregated XAPI (Windsor)|<br />
* [[Inter-domain communication for XCP|Inter-domain communication]]<br />
* [[Xenopsd|Xenops Daemon]]<br />
* [[Clipboard sharing protocol|Clipboard sharing protocol]]<br />
}}<br />
|{{Trail|Projects and Specs|<br />
* [[Juju XCP Service Provider Spec|Juju XCP Service Provider]]<br />
* [[Project Kronos]]<br />
* [[Project Zeus Fedora Spec|Project Zeus]]<br />
* [[XAPI On CentOS 6]]<br />
'''API Specs'''<br />
* [http://docs.vmd.citrix.com/XenServer/6.0.0/1.0/en_gb/api/ XAPI/XCP 1.5 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0fp1/1.0/en_gb/api/ XAPI/XCP 1.1 API spec]<br />
* [http://docs.vmd.citrix.com/XenServer/5.6.0/1.0/en_gb/api/ XAPI/XCP 1.0 API spec]<br />
'''API examples'''<br />
* [[Shutting down a VM]]<br />
}}<br />
<br>{{Trail|Previews|<br />
* [[Ceph_and_libvirt_technology_preview|XAPI + XenServer Packages + Ceph + Libvirt]]<br />
}}<br />
|{{Trail|OCAML Dev Information|<br />
* [[OCaml Best Practices for Developers|Best Practices]]<br />
* [[OCaml Coding Considerations|Coding Considerations]]<br />
* [[OCaml Cyclical Build Dependencies|Cyclical Build Dependencies]]<br />
* [[OCaml Doc : Documenting code in OCaml|Documenting Code]]<br />
* [[OCaml Editing|Editing]]<br />
* [[OCaml Errors|Errors]]<br />
* [[OCaml Installation Advice|Installation]]<br />
* [[OCaml Resources|Resources]]<br />
}}<br />
}}<br />
<br><br />
<br />
== Other Information for XAPI Developers ==<br />
=== Development Docs ===<br />
The following documents contain indexes that may be incomplete<br />
* {{Button|[[Xen_Development_Projects#Xen_Cloud_Platform_.28XCP.29_and_XAPI_projects|XAPI Development Projects]]}}<br />
* {{Button|[[XCP Design and Architecture Docs|XCP Design and Architecture Docs]]}}<br />
* {{Button|[[XAPI Developer Guide|XAPI Developer Docs]]}}<br />
<br />
<br />
{{Anchor|docs}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Disk_import/export_APIs&diff=13848Disk import/export APIs2015-01-20T16:46:20Z<p>Dave.scott: </p>
<hr />
<div>Note: official APIs have been developed, merged and documented here: http://xapi-project.github.io/xen-api/snapshots.html<br />
<br />
Note: these APIs are considered experimental and are subject to change. To provide feedback and/or query current status, email xen-api@lists.xenproject.org<br />
<br />
= What are these APIs for? =<br />
<br />
These APIs provide a safe and efficient method to import and export Virtual Disk image (VDI) data to and from a host running xapi. These APIs allow clients to efficiently backup and restore VM disks where only changed blocks are processed, rather than full disk images.<br />
<br />
These APIs are present in the following xapi versions:<br />
* TODO<br />
<br />
To detect the presence of these APIs:<br />
* TODO: either expose a capability or suggest a "try it and see" approach<br />
<br />
= Using the 'xe' CLI =<br />
<br />
== Exporting a full disk image ==<br />
<br />
The following command will export a VDI with uuid $VDI in vhd format to the local file 'full.vhd':<br />
<br />
xe vdi-export uuid=$VDI filename=full.vhd format=vhd --progress<br />
<br />
The vhd format can be *sparse*, so if there are unallocated blocks within the VDI which xapi is aware of, they will be omitted from the vhd therefore making the vhd smaller. This should work on all current vhd-based storage types (EXT, NFS)<br />
<br />
The currently supported formats are:<br />
* vhd<br />
* raw<br />
<br />
== Exporting only block changes/deltas ==<br />
<br />
Assuming you have taken a VDI snapshot with a command like:<br />
<br />
SNAPSHOT=$(xe vdi-snapshot uuid=$VDI)<br />
<br />
You can export only the changes between the VDI and the snapshot with:<br />
<br />
xe vdi-export uuid=$VDI base=$SNAPSHOT filename=delta.vhd format=vhd --progress<br />
<br />
This should be a lot quicker than a full export and the output file should be smaller.<br />
<br />
The currently supported formats are:<br />
* vhd<br />
<br />
Note the 'vhd parent' field within delta.vhd will not directly reference 'full.vhd'.<br />
<br />
== Importing a full disk image ==<br />
<br />
First create a fresh VDI to hold the data:<br />
<br />
RESTORE=$(xe vdi-create name-label=restore virtual-size=...)<br />
<br />
Then import the blocks<br />
<br />
xe vdi-import uuid=$RESTORE filename=full.vhd format=vhd --progress<br />
<br />
== Importing only changed blocks/deltas ==<br />
<br />
Import the changed blocks:<br />
<br />
xe vdi-import uuid=$RESTORE filename=delta.vhd format=vhd --progress<br />
<br />
= Using the XenAPI =<br />
<br />
The APIs are exposed as HTTP GET and HTTP PUT operations.<br />
<br />
The client can authenticate either by supplying a 'session_id=<session ref>' query parameter, or (if testing on the command-line via curl/wget) HTTP basic authentication can be used.<br />
<br />
A sophisticated client should create a Task and supply a 'task_id=<task ref>' query parameter, allowing the client to track progress and success/failure of the operation. Note the HTTP layer cannot provide application-level success/failure information as the 'HTTP 200 OK' has to be sent *before* the data has been uploaded or downloaded. Note a client which supplies its own Task is also responsible for cleaning it up afterwards with a Task.destroy.<br />
<br />
== Exporting a disk image (full or deltas) ==<br />
<br />
HTTP GET /export_raw_vdi?vdi=<VDI ref>[&session_id=<session ref>][&task_id=<task ref>][&format=<format>][&base=<base>]<br />
<br />
where<br />
* vdi: (required) the disk to export<br />
* session_id: (optional) a logged in session. If not provided then the client must use basic auth.<br />
* task_id: (optional) an existing task for tracking progress<br />
* format: (optional) the output format to use<br />
* base: (optional) if this parameter is present then only deltas will be generated.<br />
<br />
== Importing a disk image (full or deltas) ==<br />
<br />
HTTP PUT /import_raw_vdi?vdi=<VDI ref>[&session_id=<session ref>][&task_id=<task ref>][&format=<format>]<br />
<br />
where<br />
* vdi: (required) the disk to import<br />
* session_id: (optional) a logged in session. If not provided then the client must use basic auth.<br />
* task_id: (optional) an existing task for tracking progress<br />
* format: (optional) the output format to use<br />
<br />
It is not necessary to specify that the disk contains deltas-- this will be inferred from the input stream.</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Running_XCP/xapi_on_ARM&diff=12886Running XCP/xapi on ARM2014-10-08T10:49:02Z<p>Dave.scott: Created page with "This is still under development-- if you have trouble then send an email to the xen-api mailing list. You will need * a Cubieboard2 or a Cubietruck * an SD card which you can fl…"</p>
<hr />
<div>This is still under development-- if you have trouble then send an email to the xen-api mailing list.<br />
<br />
You will need<br />
* a Cubieboard2 or a Cubietruck<br />
* an SD card which you can flash<br />
<br />
First download one of the Mirage pre-built images on your development box:<br />
<br />
wget http://blobs.openmirage.org/cubieboard2.tar<br />
wget http://blobs.openmirage.org/cubietruck.tar<br />
<br />
Untar to reveal the image file itself:<br />
<br />
tar -xvf $FILE.tar<br />
<br />
Insert your SD card into your development box and discover the device name. You must find the correct device! Don't overwrite the wrong device! You can flash the SD with something like:<br />
<br />
sudo dd if=$FILE.img of=$MY_SD_DEVICE bs=1M<br />
sudo sync<br />
<br />
Eject the SD card and insert it into your Cubie. I recommend attaching the serial console but the image will also advertise itself using multicast DNS as 'cubieboard.local' or 'cubietruck.local'. Power on the cubie and wait for it to boot.<br />
<br />
Log into the cubie with mirage/mirage and add 2 new repos:<br />
<br />
cd /etc/apt/sources.list.d<br />
sudo wget http://xenbits.xenproject.org/djs/linaro-xen-4-4-talex5.list<br />
sudo wget http://xenbits.xenproject.org/djs/linaro-xapi-4-4-talex5.list<br />
sudo apt-get update<br />
<br />
Now install the xapi toolstack using apt-get:<br />
<br />
sudo apt-get install xenserver-core<br />
<br />
Unfortunately xapi (or rather xcp-networkd) doesn't like bridges being made for it, so edit your /etc/network/interfaces to read:<br />
<br />
auto lo<br />
iface lo inet loopback<br />
<br />
auto eth0<br />
iface eth0 inet dhcp<br />
up ip link set eth0 up<br />
<br />
i.e. remove the 'br0' bridge configuration.<br />
<br />
Reboot the Cubie to clear the network state. Note the current kernel has a bug causing reboot to hang. Pull the power (and the serial cable in case you're seeing leakage from it), wait a few seconds and then reinsert.<br />
<br />
Log back into the Cubie and run the setup wizard:<br />
<br />
sudo xenserver-install-wizard<br />
<br />
Be patient while xapi starts -- if you see some errors on the terminal don't worry. If it fails then run it again: it should be idempotent. If it still doesn't work then email the list.<br />
<br />
Now you should be able to run 'sudo xe' and connect XenCenter!</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=Category:XCP&diff=12885Category:XCP2014-10-08T10:38:03Z<p>Dave.scott: </p>
<hr />
<div>'''Go [http://wiki.xenserver.org/Category:XCP here] to see all documents for XCP!'''<br><br />
<hr><br><br />
{{TODO|This category needs to be split into 3 categories as per [http://lists.xenproject.org/archives/html/xen-api/2013-06/msg00109.html this proposal] (check the responses too):<br />
* {{Cl|XAPI Devel}} : Documents relevant for developers of the XAPI Project {{HalfDone}}<br />
* {{Cl|XAPI Users}} : Documents for users of software created by the XAPI Project that apply to <br />
** <em>all</em> variants of XAPI (XAPI Packages, XCP, XenServer, ...). For example API documentation, command line documentation, etc.<br />
** apply to XAPI packages in Linux distributions (maybe a sub-category would be appropriate) <br />
* {{Cl|XCP}} : Documents for users and developers of XCP that <em>only</em> apply to XCP<br />
Content on this page needs to be reviewed and assigned to the appropriate categories.<br />
}}<br />
<br />
Xen Cloud Platform (XCP) - an open source project, providing a bare-metal installable ISO with a full Xen hypervisor virtualization stack. It was initially an open source version of XenServer, but has slowly developed into its own system. Note that '''XCP is a full binary distribution of Xen''', whereas the Xen hypervisor needs to be built from source!<br />
<br />
== Information for XCP Users ==<br />
{{Trailbox|Getting Started with XCP|<br />
|{{Trail|Introduction|<br />
* [[XCP_Overview|Overview]] <br />
* [http://xen.org/download/xcp/index.html Download XCP]<br />
* [[XCP toolstack on a Debian-based distribution|XCP on a Debian-based distro]]<br />
* [[Video: Introduction to Virtualization Xen, XCP, and the Cloud|Intro to Xen, XCP & the Cloud]]<br />
}}<br />
|{{Trail|Features|<br />
* [[XCP Release Features]]<br />
* [[XCP/XenServer Feature Matrix|XCP vs. XenServer Features]]<br />
* [[Xen / XCP / XCP on Linux Overview|Xen vs. XCP vs. XCP on Linux]]<br />
}}<br />
|{{Trail|Choices|<br />
* [[Choice of Toolstacks]]<br />
* [[XCP Management Tools]]<br />
* [[Using XenCenter to manage XCP|Using XenCenter with XCP]]<br />
* [[XAPI_on_Ubuntu|Try XCP on Ubuntu]]<br />
}}<br />
|{{Trail|Get Help|<br />
* [http://xen.org/support/community.html Getting help!] <br />
* [http://xen.org/community/projects.html Get paid help!]<br />
* [[Reporting Bugs against XCP|Report Bugs]]<br />
* [http://www.xen.org/products/xcp/community_and_support.html XCP Community Resources] <br />
* '''IRC Freenode:''' #xen-api<br />
* '''Mail:''' xen-api@lists.xen.org <br />
}}<br />
}}<br />
<br />
{{Trailbox|XCP Overviews, Manuals, ...|<br />
|{{Trail|Guest Install|<br />
* [[CentOS 6 Virtual Machine installation on XCP|CentOS 6]]<br />
* [[FreeBSD 9.0 64-bit HVM on XCP 1.1|FreeBSD 9.0 (64-bit)]]<br />
* [[OpenSuse 11.4 VM (64 bit) interactive installation on XCP|OpenSuse 11.4 (64 bit)]]<br />
* [[How to create custom XCP templates|Create custom XCP templates]]<br />
}}<br>{{Trail|Other Installs|<br />
* [[Installing the XCP CLI (XE)]]<br />
}}<br />
|{{Trail|Overviews and Manuals|<br />
* [[XAPI|XAPI Overview]]<br />
'''Manuals''' <br />
* [[XCP 1.0 Manuals]]<br />
* [[XCP 1.1 Manuals]]<br />
* [[XCP 1.5 Manuals]]<br />
* [[:Category:Manual#XCP|All XCP Manuals]]<br />
}}<br>{{Trail|Other Documents|<br />
* [[XCP Building Instructions]]<br />
* [[Repacking the XCP ISO]]<br />
* [[XCP Command Line Interface]]<br />
* [[Open vSwitch|Introduction to Open vSwitch]]<br />
}}<br />
|{{Trail|HowTo's and Tutorials|<br />
* [[XCP_HowTo|HowTo's Index]]<br />
* [[Xen and XCP Tutorial|Introduction to Xen and XCP]]<br />
* [[XCP: From the Enterprise Data Center to the Cloud]]<br />
* [[XCP: Nuts and Bolts]]<br />
}}<br />
|{{Trail|Specialist Topics|<br />
* [[Xen Cloud Platform: Access to VM console|Access to VM Consoles]]<br />
* [[Creating an iSCSI target on XCP 1.1|Creating an iSCSI target]]<br />
* [[XCP DirectDiskAccess|Direct Disk Access]]<br />
* [[Network Throughput and Performance Guide|Network & Performance Guide]]<br />
* [[XCP as virtual machine|Running XCP in a Xen VM]]<br />
* [[XCP and VLANs|VLANs]]<br />
}}<br />
}}<br />
<br />
{{Trailbox|XCP Examples, FAQs, Cloud, ...|<br />
|{{Trail|Examples and Helper Scripts|<br />
* [[Helper files and scripts for XCP]]<br />
* [[Networking with the XCP toolstack|Networking with XCP]]<br />
* [[Running XCP/xapi on ARM]]<br />
}}<br />
|{{Trail|FAQs|<br />
* [[XCP FAQ General|General Questions]]<br />
* [[XCP FAQ Dynamic Memory Control|Dynamic Memory Control]]<br />
* [http://serverfault.com/questions/tagged/xenserver XCP on serverfault]<br />
}}<br />
|{{Trail|Cloud and XCP|<br />
* [[XCP CloudStack|CloudStack]]<br />
* [[OpenNebulaXCP|OpenNebula]]<br />
}}<br />
|{{Trail|OpenStack|<br />
* [[XCP OpenStack|OpenStack]]<br />
* [[XCP DevStack|DevStack]]<br />
* [[Video: Tutorial for installing Openstack with XenServer|Video Tutorial]]<br />
}}<br />
}}<br />
<br />
== Information for XAPI Developers ==<br />
{{Morebar|XAPI Developer Documentation|[[:Category:XAPI Devel|More]]}}</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12620VAAI support2014-09-15T13:02:10Z<p>Dave.scott: </p>
<hr />
<div>We want to speed up common VM disk operations such as:<br />
# deleting disks<br />
# creating empty disks<br />
# cloning disks<br />
Ideally the host would send the commands to the storage server/target and allow it to Do The Right Thing. Many arrays are 'smart' in the sense that they maintain block allocation maps and can create trees of disk clones without explicitly copying all the blocks. Even when an array doesn't have such smarts it is still better to copy blocks across the internal array fabric than across the host to array network link.<br />
<br />
The VMware vStorage APIs for Array Integration (VAAI) are a defacto standard which allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
How can we take advantage of storage arrays which support these?<br />
<br />
= Notes on VAAI =<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]]<br />
* Linux 3.16-rc4 has a fixed EXTENDED COPY which is enough for a proof-of-concept [[device mapper implementation (Jul 2014)|https://www.redhat.com/archives/dm-devel/2014-July/msg00066.html]]. This is automatically used if you use mirror, snapshot, thin or cache dm targets.<br />
* Windows uses ODX (Offloaded Data Transfer) rather than EXTENDED COPY<br />
* A comparison of file vs block primitives: http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/<br />
* A draft of NFS v4.2 has [[server-side copy|https://tools.ietf.org/html/draft-ietf-nfsv4-minorversion2-26#section-1.4.1]]</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12619VAAI support2014-09-15T12:59:45Z<p>Dave.scott: </p>
<hr />
<div>We want to speed up common VM disk operations such as:<br />
# deleting disks<br />
# creating empty disks<br />
# cloning disks<br />
Ideally the host would send the commands to the storage server/target and allow it to Do The Right Thing. Many arrays are 'smart' in the sense that they maintain block allocation maps and can create trees of disk clones without explicitly copying all the blocks. Even when an array doesn't have such smarts it is still better to copy blocks across the internal array fabric than across the host to array network link.<br />
<br />
The VMware vStorage APIs for Array Integration (VAAI) are a defacto standard which allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
How can we take advantage of storage arrays which support these?<br />
<br />
= Notes on VAAI =<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]]<br />
* Linux 3.16-rc4 has a fixed EXTENDED COPY which is enough for a proof-of-concept [[device mapper implementation (Jul 2014)|https://www.redhat.com/archives/dm-devel/2014-July/msg00066.html]]. This is automatically used if you use mirror, snapshot, thin or cache dm targets.<br />
* Windows uses ODX (Offloaded Data Transfer) rather than EXTENDED COPY<br />
* A comparison of file vs block primitives: http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12618VAAI support2014-09-15T12:56:10Z<p>Dave.scott: </p>
<hr />
<div>We want to speed up common VM disk operations such as:<br />
# deleting disks<br />
# creating empty disks<br />
# cloning disks<br />
Ideally the host would send the commands to the storage server/target and allow it to Do The Right Thing. Many arrays are 'smart' in the sense that they maintain block allocation maps and can create trees of disk clones without explicitly copying all the blocks. Even when an array doesn't have such smarts it is still better to copy blocks across the internal array fabric than across the host to array network link.<br />
<br />
The VMware vStorage APIs for Array Integration (VAAI) are a defacto standard which allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
How can we take advantage of storage arrays which support these?<br />
<br />
= Notes on VAAI =<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]]<br />
* Linux 3.16-rc4 has a fixed EXTENDED COPY which is enough for a proof-of-concept [[device mapper implementation (Jul 2014)|https://www.redhat.com/archives/dm-devel/2014-July/msg00066.html]]. This is automatically used if you use mirror, snapshot, thin or cache dm targets.<br />
* A comparison of file vs block primitives: http://cormachogan.com/2012/11/08/vaai-comparison-block-versus-nas/</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12617VAAI support2014-09-15T12:52:17Z<p>Dave.scott: </p>
<hr />
<div>We want to speed up common VM disk operations such as:<br />
# deleting disks<br />
# creating empty disks<br />
# cloning disks<br />
Ideally the host would send the commands to the storage server/target and allow it to Do The Right Thing. Many arrays are 'smart' in the sense that they maintain block allocation maps and can create trees of disk clones without explicitly copying all the blocks. Even when an array doesn't have such smarts it is still better to copy blocks across the internal array fabric than across the host to array network link.<br />
<br />
The VMware vStorage APIs for Array Integration (VAAI) are a defacto standard which allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
How can we take advantage of storage arrays which support these?<br />
<br />
= Notes on VAAI =<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]]<br />
* Linux 3.16-rc4 has a fixed EXTENDED COPY which is enough for a proof-of-concept [[device mapper implementation (Jul 2014)|https://www.redhat.com/archives/dm-devel/2014-July/msg00066.html]]. This is automatically used if you use mirror, snapshot, thin or cache dm targets.</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12616VAAI support2014-09-15T12:40:29Z<p>Dave.scott: </p>
<hr />
<div>The VMware vStorage APIs for Array Integration (VAAI) allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]]<br />
* Linux 3.16-rc4 has a fixed EXTENDED COPY which is enough for a proof-of-concept [[device mapper implementation (Jul 2014)|https://www.redhat.com/archives/dm-devel/2014-July/msg00066.html]]. This is automatically used if you use mirror, snapshot, thin or cache dm targets.</div>Dave.scotthttps://wiki.xenproject.org/index.php?title=VAAI_support&diff=12615VAAI support2014-09-15T12:36:44Z<p>Dave.scott: </p>
<hr />
<div>The VMware vStorage APIs for Array Integration (VAAI) allow common VM disk operations to be sped up including<br />
# Atomic Test and Set (ATC): for fast LV locking<br />
# Block-level Zero: for blanking freshly-created LVs<br />
# Block-level Clone: for getting the array to copy LVs<br />
# Block-level Delete: to clear unused space<br />
<br />
(for more information see http://linux-iscsi.org/wiki/VStorage_APIs_for_Array_Integration )<br />
<br />
The APIs are presented as SCSI commands and have been implemented in the [[Linux-IO software target|http://linux-iscsi.org/wiki/LIO]] which has achieved VMware certification in several hardware products.<br />
<br />
* Linux has a BLKZEROOUT ioctl<br />
* Linux 3.12 has [[EXTENDED COPY|http://linux-kernel.2935.n7.nabble.com/PATCH-0-9-target-Add-support-for-EXTENDED-COPY-VAAI-offload-emulation-td707824.html]] which you can run via [[sg_xcopy|http://manpages.ubuntu.com/manpages/saucy/man8/sg_xcopy.8.html]] and which has been added to [[device mapper in Jul 2014|https://www.redhat.com/archives/dm-devel/2014-July/msg00070.html]]</div>Dave.scott