There are several options to publish data using semantic standards. For many website owners, the easiest is to embed the extra markup directly into the HTML pages by using RDFa or Microformats. In this way the same page can read by people and parsed by computers; for most publishers and content management systems this simply means adding the markup support into the current HTML templates.

An alternative to embedding semantic data into your pages is to publish two formats of the content: one format for people, and a different one computers. This is done by supporting content negotiation as described in the tutorial How to publish Linked Data on the Web.  In supporting semantic content negotiation, a web server which normally provides HTML sends RDF format if the HTTP request Accept header contains 'application/rdf+xml'. Sindice uses an Accept header similar to application/rdf+xml, text/turtle, text/n3, application/xhtml+xml; q=0.9, text/html; q=0.8, text/plain; q=0.6, */*. Content negotiation allows people to view the standard HTML pages, while semantic engines like Sindice receive the corresponding semantic data directly.

Content providers who wish simply to provide a large RDF dataset dump can do so by following the procedures defined for the Semantic Sitemap Extension. While this format is less useful for frequently changing data, it is sometimes an appropriate choice for large datasets.

Sindice, the semantic web index at www.sindice.com supports all of the above methods. 

For help in publishing great data, be sure you visit the Sindice Web Data Inspector. The Web Data Inspector will assist you by providing interactive data visualization and validation services.

Also you can send notification of changes directly to Sindice using our PING interface. See section 2.4 below.

Whether you publish your semantic content using a standard sitemap, an RSS or Atom feed, or a semantic sitemap, be sure to list the sitemap in your robots.txt file. A site's robot.txt is the first place Sindice and many other engines will look when trying to discover your site. Add a line like the following into your robots.txt file:

Sitemap: http://www.example.com/sitemap.xml

2.1 Include time indications in your Sitemap

Sitemaps are the standard way to let crawlers know about the pages on your website. When sitemaps provide time indications using lastmod, changefreq and priority fields, they can be used effectively to have Sindice and others download only new and changed pages.

Sitemaps are usually named "sitemap.xml" and should be advertised in the server's robots.txt file. The sitemap lists the URLs of your pages which either containing embedded metadata or are supported by content negotiation. Since the sitemaps will also be used also by conventional HTML crawlers they should not include direct links to pure RDF representation of your pages.

• Sindice uses the lastmod field to decide if the given URL has to be re-indexed or not. This is probably the most important information, as this can reduce the number of requests Sindice will to make to your site. If you provide lastmod for every listed URL you can expect Sindice to fetch only the sitemap.xml file (e.g. daily) and then fetch only those URLs which have been modified.
• The changefreq field is used to determine how often the sitemap containing the given URL needs to be fetched. The sitemap will be fetched with the highest frequency indicated by the URLs contained in it. Because of this, it can save bandwidth if items with the same change frequency are grouped into separate sitemaps. When a lastmod is not available it is used to decide if the page is to be fetched or not.
  
• The priority field determines priority of this URI relative to other URIs on your site. This is a number between 0 and 1, and it helps Sindice to decided which pages should be fetched if it has not enough resources to fetch them all. The default priority is 0.5, and higher priority pages will get more attention from Sindice (and other indexing engines).

Sindice supports sitemap indexes as well as compressed sitemaps. Using compression is strongly suggested for large sitemaps. Refer to the the sitemap standard for full detail at www.sitemaps.org.

2.2 Provide RSS or Atom feeds

RSS and Atom web feeds are a standard way of publishing the list of pages recently changed on your site. If you have a web feed which points to content containing embedded metadata or supporting content negotiation, Sindice can follow that feed with no further changes required. To have Sindice follow your RSS feeds, list them as sitemaps in your robots.txt file. Sindice will fetch such RSS feeds as frequently as each hour, and will index the new pages there listed.

Sitemap: http://www.example.com/mysite.rss

2.3 Using Semantic sitemaps

A semantic sitemap is a sitemap extension to be used when your Web site provides dataset DUMPS (e.g. the content as a single RDF file) and to expose further semantic service such as for example SPARQL endpoints.  To have Sindice index your whole dataset this is the best option. Semantic sitemaps should be advertised in the server's robots.txt file exactly as plain old sitemaps should.

A simple example is the following.

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9"
        xmlns:sc="http://sw.deri.org/2007/07/sitemapextension/scschema.xsd">
  <sc:dataset>
    <sc:datasetLabel>Example Corp. Product Catalog</sc:datasetLabel>
    <sc:datasetURI>http://example.com/catalog.rdf#catalog</sc:datasetURI>

2.4 Sending automatic notification to Sindice

3. Tell Sindice about your site

Once you are ready with any of the above methods, simply use the online form to send a link of your sitemap or to PING us any page containing embedded metadata or supporting content negotiation.

Once Sindice has discovered your sitemap we will do a sample analysis of your site (up to 1000 pages) to determine whether you are publishing effectively into the semantic web of data. If you are, Sindice will begin the synchronization process. 

4. Check that your site has been Discovered and Synchronized

If you have comments, questions or encounter problems publishing into the web of data for effective discovery and synchronization, we'd love to hear from you at the Sindice Developers discussion group. Find us at http://groups.google.com/group/sindice-dev/topics.