Should I use XML or text, JavaScript, or HTML as a return type?

It depends. Clearly the 'X' in AJAX stands for XML, but several AJAX proponents are quick to point out that nothing in AJAX, per se, precludes using other types of payload, such as, JavaScript, HTML, or plain text.

* XML - Web Services and AJAX seem made for one another. You can use client-side API's for downloading and parsing the XML content from RESTful Web Services. (However be mindful with some SOAP based Web Services architectures the payloads can get quite large and complex, and therefore may be inappropriate with AJAX techniqes.)
* Plain Text - In this case server-generated text may be injected into a document or evaluated by client-side logic.
* JavaScript - This is an extension to the plain text case with the exception that a server-side component passes a fragment of JavaScript including JavaScript object declarations. Using the JavaScript eval() function you can then create the objects on the client. JavaScript Object Notation (JSON), which is a JavaScript object based data exchange specification, relies on this technique.
* HTML - Injecting server-generated HTML fragments directly into a document is generally a very effective AJAX technique. However, it can be complicated keeping the server-side component in sync with what is displayed on the client.

Mashup is a popular term for creating a completely new web application by combining the content from disparate Web Services and other online API's. A good example of a mashup is housingmaps.com which graphically combines housing want-ads from craiglist.org and maps from maps.google.com.

Are there Usability Issues with AJAX?

The nature of updating a page dynamically using data retrieved via AJAX interactions and DHTML may result in drastically changing the appearance and state of a page. A user might choose to use the browser's back or forward buttons, bookmark a page, copy the URL from the URL bar and share it with a friend via an email or chat client, or print a page at any given time. When designing an AJAX based application you need to consider what the expected behavior would be in the case of navigation, bookmarking, printing, and browser support as described below.

* Navigation - What would be the expected behavior of the back, forward, refresh, and bookmark browser buttons in your application design. While you could implement history manipulation manually it may be easer to use a JavaScript frameworks such as Dojo that provides API's history manipulation and navigation control.
* Bookmarking and URL sharing - Many users want to bookmark or cut and paste the URL from the browser bar. Dojo provides client-side for bookmarking and URL manipulation.
* Printing - In some cases printing dynamically rendered pages can be problematic.
Other considerations as a developer when using AJAX are:
* Browser Support - Not all AJAX/DHTML features are supported on all browsers or all versions of a browser. See quirksmode.org for a list of browser support and possible workarounds.
* JavaScript disabled - You should also consider what happens if the user disables JavaScript. Additionally, there are several legitimate reasons why JavaScript and CSS support may be unavailable on a user's web browser.
* Latency - Keep in mind latency in your design. A running application will be much more responsive than when it is deployed.
Latency problems: myth or reality?
* Accessibility - Guaranteeing your site is accessible to people with disabilities is not only a noble goal, it is also requited by law in many markets. Some marvelous enabling technology is available to help people use the Web in spite of disabilities including visual, auditory, physical, speech, cognitive, and neurological disabilities. With a little forethought, and comprehension of some well documented best practices, you can assure that your application is compatible with that enabling technology.
Degradability is the term used to describe techniques used by web applications to adapt to the wide range of web browser capabilities. Many AJAX libraries have automatic degradability built in. But if you are coding your own custom AJAX functionality, simply taking some care to follow the best practices promoted by standards bodies like the World Wide Web Consortium (W3C), and grass root movements like the Web Standards community and many others, your application can run usefully on browsers that are incapable of AJAX behaviors. Granted, your application may loose some of the "wow factor" on these less capable browsers, but your application will still be usable.
Remember to not design with AJAX just for the sake of coolness. The reason you built your application is so people will use it. And people will not use your application if your application is not compatible with their web browser.